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Abstract 

How  to  better  evaluate  the  merits  of  competing  technological  concepts. 

This  project  is  concerned  with  evaluation  of  the  merits  of  competing  technological 
concepts  and  managing  the  R&D  investments  needed  to  bring  them  to  fruition.  A 
weakness  of  the  current  R&D  process  is  that  arguments  regarding  who  should  receive 
funding  come  from  a  concept’s  proponents,  who  are  usually  interested  and  biased.  More 
objective  evaluation  methods  are  needed.  As  part  of  this  work  it  is  important  to 
understand  better  how  qualified  experts  evaluate  technological  concepts.  Can  a 
probabilistically  formulated  method  of  integrating  knowledge  of  various  performance 
attributes  provide  better  understanding  of  the  likely  performance  of  a  technological 
concept?  This  is  the  question  of  interest. 

A  nuclear  power  plant  example  (impetus  for  the  actual  study  below). 

The  impetus  for  the  study  began  with  The  U.S.  Department  of  Energy’s  Generation  TV 
advanced  reactor  technology  program,  the  program  that  will  select  the  next  generation  of 
nuclear  reactors.  Generation  IV  chose  twenty-seven  criteria  for  use  in  determining  which 
nuclear  power  plant  concepts  would  be  best  for  a  given  mission.  These  criteria  came  in 
the  form  of  twenty-seven  questions  asked  of  prospective  concept  designers.  The  concept 
designers  ranked  their  own  design  over  a  range  of  seven  bins  and  specified  a  peak  in  the 
most  likely  bin.  The  27  criteria  were  assumed  to  be  independent  and  were  used  in 
creating  three  major  goals  (sustainability,  safety  &  reliability,  and  economics).  That  is, 
the  score  assigned  in  each  of  the  27  areas  was  rolled  into  3  major  scores  called  goals  in 
this  study.  Weights,  unknown  to  the  concept  designers,  were  assigned  to  individual 
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questions  and  the  three  major  goals,  and  then  probability  mass  functions  were  created 
predicting  the  success  of  a  given  design. 

A  robot  design  course. 

At  the  Massachusetts  Institute  of  Technology  (MTT)  between  Fall  semester  and  Spring 
semester  (i.e.,  during  January)  an  independent  activities  period  (IAP)  offers  the 
opportunity  for  students  to  design  robots  in  MASlab  (Mobile  Autonomous  Systems 
Laboratory)  Robotics  Competition,  also  known  as  course  6.186.  Therefore,  course  6.186 
provides  an  opportunity  for  evaluating  technological  concepts  (i.e.,  in  the  form  of  a  robot 
design  as  well  as  operational  contests  of  those  designs).  Course  6.186  provides  an 
opportunity  for  students  to  act  as  consultants  in  offering  their  expertise  in  the  evaluation 
of  robots  designed  by  themselves  and  their  competitors.  The  evaluations  are  composed 
of  questionnaires  similar  to  those  described  in  the  nuclear  power  example.  The 
consultants’  responsibilities  are  to  evaluate  the  strengths  and  weaknesses  of  other  robots. 
The  quality  of  the  evaluations  is  indicated  by  the  results  of  the  robot  competition. 


From  this  experience  we  learn  more  about  how  objective  evaluations  of  the  performance 
of  competing  concepts  can  be  made.  As  coordinators  of  this  effort,  we  identify  the 
methodology  of  those  consultants  who  were  most  successful  in  identifying,  before 
testing,  the  best  robot  designs.  The  methodologies  thus  identified  can  be  extended  to 
large-scale  projects  in  general  such  as  identification  of  the  best,  among  competing, 
technological  concepts. 
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Chapter  1 


Introduction 

1.1  Literature  search 

Two  examples  were  covered  in  the  literature  search,  which  follows  below.  The  two 
examples  are  the  Navy  DD  (X)  acquisition  process  and  how  technology  investments  are 
targeted  in  the  drug  delivery  process.  Both  examples  provide  insight  into  the 
methodology  used  in  selection  of  technological  concepts  for  R&D  support,  and, 
therefore,  were  deemed  useful  to  our  project. 

1.1.1  Navy  DD  (X)  acquisition  process  example  [1] 

Because  of  the  large  expense  in  designing  and  producing  a  naval  vessel  (i.e.,  on 
the  order  of  billions  Of  dollars),  the  R  &  D  process  leading  to  the  detailed  design  and 
construction  is  unique  to  this  industry.  First,  the  two  major  corporations  owning 
shipyards  in  the  United  States,  Northrop  Grumman  and  General  Dynamics,  created  an 
alliance  to  ensure  that  both  shipyards  would  participate  in  the  construction  of  the  DD  (X), 
thus  ensuring  the  competitive  capability  of  both  corporations’  shipyards.  A  competition 
followed  between  the  two  corporations,  called  teams,  to  obtain  what  the  Navy  calls  the 
DD  (X)  design  agent  contract  which  includes  the  design  of  the  ship  and  building  and 
testing  of  engineering  development  models  (i.e.,  prototypes  of  new  systems). 

The  above  process  is  part  of  a  five-phase  process  of  which  the  first  two  phases  are 
described  in  this  paragraph.  Phase  I  involved  both  teams  in  ship  design  and  support 
concepts,  performance  specification,  and  system  and  support  requirement  determination. 
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Phase  II  involved  both  teams  in  preliminary  ship  and  support  design  and  initial  system 
specification. 

Phase  III  is  the  point  at  which  down-selection  to  one  design  occurs,  which 
occurred  on  29  April  2002  when  Northrop  Grumman  became  the  design  agent.  Phase  HI 
is  when  the  engineering  development  models  will  be  built  and  tested.  Phase  HI  also 
includes  production  readiness,  complete  system  and  support  design,  and  critical  design. 

It  Is  worth  noting  again,  that  although  one  team’s  design  was  selected,  this  does  not 
exclude  the  other  team.  “Both  design  teams  come  together  as  a  single  team  and 
participate  in  the  design  of  the  ship,  so  that  both  yards  are  prepared  to  build  the  ship  when 
it  comes  time  to  have  the  competition  in  2005.  ” 

The  competition  spoken  of  above  is  that  for  the  lead  ship  contract  and  leads  to 
Phase  IV.  “We  expect  to  have  a  second  competition  or  the  next-step  competition  to 
award  detailed  design  and  final  construction  of  the.  lead  ship.”  Again,  both  teams  will 
participate  once  the  competition  has  been  decided  except  that  the  “lead  designer”  will,  by 
definition,  have  the  lead  seat  at  the  table  in  making  decisions  about  the  design.  Finally, 
Phase  V  occurs  which  is  simply  the  execution  of  engineering  and  logistics  life  cycle 
support  of  the  DD  (X). 

1.1.2  Targeting  technology  investments  in  the  drug  delivery  process  [  2] 

The  acquisition  process  for  a  naval  vessel  is  described  above,  offering  insight  into 
the  steps  the  Navy  must  follow  to  target  its  technology  investments  (i.e.,  the  selection  of 
new  vessel  designs  and  the  subsequent  construction  of  these  new  vessels). 
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Pharmaceutical  companies  have  similar  hurdles  in  targeting  their  own  technolgy 
investments  in  the  drug  delivery  process. 

In  1990,  the  field  of  genomics  (the  study  of  DNA)  was  created.  “Comparing 
DNA  sequences  from  a  number  of  species  enables  scientists  to  study  the  function  of  a 
gene  as  it  is  expressed  in  different  organisms.  Similarities  in  gene  function  between  these 
organisms  and  humans  enable  pharmaceutical  researchers  to  use  these  organisms  to 
analyze  the  effect  of  new  therapeutic  agents  on  the  biological  function  of  specific 
proteins,  i.e.,  target  molecules.”  The  huge  volume  of  genomic  data  made  the  creation  of 
a  new  field  known  as  bioinformatics  essential.  Bioinformatics  is  used  to  extract  genomic 
data  from  public  and  private  databases  via  programs  written  to  better  understand 
biological  processes.  “Bioinformatics,  linked  to  other  new  technologies,  combinatorial 
chemistry  and  high  throughput  screening,  has  created  a  new  paradigm  for  drug  discovery. 
This  technological  change  will  move  the  industry  from  serendipitous  discovery  of  new 
drugs  to  strategic  management  of  markets  and  technology  to  improve  healthcare  for 
targeted  diseases.” 

‘Technology  management  is  critical  to  the  pharmaceutical  industry  for  a  number 
of  reasons.  First,  increased  investment  in  pharmaceutical  R&D  has  not  resulted  in  a 
significant  change  in  the  number  of  new  pharmaceutical  agents  introduced  into  the 
market.  The  cycle  time  for  development  of  hew  drugs  and  R&D  dollars  spent  per 
product  has  increased.  This  lack  of  R&D  productivity  is  caused  by  many  problems  but 
among  them  are  understanding  the  limitations  of  new  drug  discovery  technologies  (e.g., 
combinatorial  chemistry,  assays  to  validate  targets),  the  unpredictable  complexity  of 
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biological  systems,  and  difficulty  in  making  appropriate  investments  in  technological 
areas  that  increase  research  productivity.” 

“Second,  many  of  the  biotech  companies  that  provide  services  to  the 
pharmaceutical  industry  have  moved  to  internal  development  of  their  own  drugs  either 
alone  or  in  partnership  with  established  companies.  In  a  recent  survey,  it  was  estimated 
that  approximately  50  percent  of  the  drugs  sold  by  the  pharmaceutical  industry  were 
licensed  in  from  other  firms.” 

To  increase  research  productivity  at  the  corporate  level,  a  process  known  as 
roadmapping  can  be  applied.  Roadmapping  is  a  technology  management  tool  that  can  be 
used  for  “assessing  the  potential  value  of  new  technologies  in  meeting  the  challenges  of 
the  drug  discovery  process.”  Roadmapping  ensures  that  the  right  technological 
capabilities  are  in  place  to  obtain  the  desired  result.  This  presupposes  that  the  technology 
is  not  only  available  but  also  aligned  to  meet  the  desired  need.  Of  note  is  the  fact  that  a 
roadmap,  on  the  corporate  level,  focuses  on  “improving  internal  processes  which  may 
need  improvement  to  increase  R&D  productivity  or  to  upgrade  a  step  in  the  drug 
discovery  process  that  has  fallen  behind  ‘industry  standards.’  The  roadmapping  process 
can  be  divided  into  five  steps:  Team  Formation,  Focus,  Technology/Workflow  Analysis, 
Implementation,  and  Review.” 

Team  formation  ensures  the  inclusion  of  the  appropriate  people  on  the  team: 
people  from  R&D  and  technology  management,  from  business  development,  from 
finance,  from  medicinal  chemistry,  high  throughput  screening,  regulatory,  and  safety 
studies.  Aside  from  inviting  the  appropriate  people,  “the  first  priority  is  to  establish  a 
common  understanding  of  the  process  and  the  terminology  employed  in  the  analysis. 
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Taking  this  step  will  minimize  the  confusion  that  can  potentially  arise  during  the 
roadmapping  process.” 

The  focus  step  is  where  the  team  begins  the  development  of  a  “detailed  analysis 
of  the  drug  discovery  process.”  In  this  analysis  technology  is  an  important  part.  The 
introduction  of  new  technology  is  valued  provided  the  process  is,  thus,  made  more 
efficient  and  effective.  “One  approach  to  develop  a  systematic  analysis  is  to  apply 
Goldratt's  Theory  of  Constraints”  in  identifying  constraints  and  weaknesses. 

“In  the  drug  discovery  process,  application  of  this  theory  must  be  modified  based 
on  limitation  of  technology  and  knowledge  of  biological  systems.  In  the  process  of 
performing  this  analysis,  the  team  must  decide  the  metrics/  factors  required  to  evaluate 
each  step  in  the  process.  Issues  to  consider  are  costs,  predictability  of  outcome,  internal 
competencies  in  the  organization,  and  opportunities  for  technology  improvement.  Upon 
completion  of  the  analysis,  a  model  can  be  created  to  identify  steps  in  the  process  that 
have  the  greatest  potential  impact  on  increasing  research  productivity.” 

The  input  to  the  technology/workflow  analysis  is  the  deliverable  from  the  focus 
step.  Where  the  deliverable  from  the  focus  step  is  the  “identification  of  a  specific  step  in 
the  process  for  improvement  based  on  technology  availability  and  the  probability  of 
obtaining  a  successful  outcome.  The  value  of  technologies  in  the  specific  process  step  is 
dependent  on  the  degree  of  their  alignment  with  the  needs  of  the  process.  Rigorous 
evaluation  of  this  alignment  must  be  performed  to  understand  the  limitations  and  benefits 
of  the  technology.” 

A  useful  tool  in  employing  the  appropriate  technology  is  to  use  a  matrix  with  the 
process  needs  on  the  left  side  of  the  matrix  and  potential  technologies  listed  at  the  top. 
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The  technologies  will  be  ranked  either  high,  medium  or  low  depending  on  how  well 
aligned  the  technologies  are  with  respect  to  the  process  needs. 

Following  selection  of  the  appropriate  technologies,  implementation  requirements 
must  be  identified  and  forwarded  to  management.  Management  ensures  that  the 
appropriate  resources,  both  budgetary  and  personnel,  are  made  available.  “One  of  the 
benefits  of  the  roadmapping  process  is  the  higher  probability  that  implementation  goes 
according  to  plan,  since  multiple  functional  areas  were  involved  in  the  formulation  of  the 
roadmap  and  provided  expert  input  to  ensure  its  success.” 

The  review  step  then  follows  where  “any  systematic  improvement  to  a  process 
requires  that  the  team  members  learn  whether  the  process  modification  led  to  the  desired 
outcome  and,  if  appropriate,  take  further  corrective  actions.  The  Deming  Cycle  (plan-do- 
check-act)  is  a  useful  tool  for  reviewing  the  results  from  any  process  improvement.  The 
check  step  permits  the  review  of  the  agreed-upon  metrics  to  determine  whether  they  have 
met  the  goals  of  the  roadmapping  effort.  If  modifications  are  required,  the  appropriate 
members  of  the  team  should  meet  and  define  a  revised  action  plan.  In  some  cases,  this 
may  require  modifying  an  existing  technology/process  or  finding  a  new  technology,” 


1.2  Background 

1.2.1  Nuclear  power  plant  example 

The  U.S.  Department  of  Energy  Generation  IV  Reactor  Program  is  responsible  for 
selecting  and  funding  initial  development  of  the  next  generation  of  nuclear  power  plants. 
This  program  intended  to  rank  its  choices  for  selection  based  upon  scores  assigned 
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quantitatively  to  each  of  the  power  plant  concepts.  The  process  chosen  used  a  scoring 
model. 

The  scoring  process  begins  with  questionnaires  filled  out  by  the  concept  designers 
for  each  of  the  nuclear  power  plants.  Each  concept  designer  answers  questions  regarding 
the  projected  success  of  the  concept  with  respect  to  a  set  of  performance  goals.  The 
questions  asked  allowed  the  designers  to  select  the  range  of  success  in  the  area  in 
question  and  the  potential  peak  or  most  likely  degree  of  success.  In  this  study,  seven  bins 
(or  discrete  scores)  comprised  the  entire  range  from  which  a  score  could  be  selected,  with 
bin  one  being  the  lowest  score  and  bin  seven  the  highest. 

The  Generation  IV  program  assigned  probability  mass  functions  based  upon  the 
range  and  peak  selected  by  the  designers  for  each  question.  For  example,  if  a  concept 
designer  selected  bins  3,4,  and  5  with  a  peak  selected  at  4  for  a  specific  question,  then  the 
Generation  IV  might  assign  a  probability  mass  function  of  0.2  for  bin  3, 0.6  for  bin  4  and 
0.2  for  bin  5.  This  same  process  was  followed  for  each  of  the  questions.  Each  of  the 
questions  fit  into  one  of  three  major  sub-goals.  Weightings  (with  values  unknown  by  the 
concept  designers)  were  assigned  to  each  of  the  specific  questions  allowing  probability 
mass  functions  to  be  calculated  for  each  of  the  three  major  goals.  Weighting  were  also 
assigned  to  each  of  the  three  major  sub-goals  allowing  an  overall  probability  mass 
function  to  be  calculated  for  each  of  the  nuclear  power  plant  concepts. 

1.2.2  Reasons  for  utilization  of  data  from  the  robot  design  contest 

Potential  problems  with  the  above  process  exist.  First,  the  work  conducted  by  the 
Generation  IV  program  was  purely  hypothetical  (i.e.,  none  of  the  nuclear  power  plant 
concepts  has  actually  been  built).  Secondly,  because  none  of  the  power  plants  was  built. 
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this  means  that  none  could  be  compared  operationally  against  their  concept  designer’s 
projections  of  success  (as  shown  in  the  probability  mass  functions  created  via  the 
questionnaire).  Third,  because  the  concept  designers  themselves  critiqued  their  own 
design,  inherent  bias  exists.  This  is  obvious  due  to  the  fact  that  research  and 
development  money,  followed  by  the  likely  prospect  of  construction  of  the  power  plant 
would  be  the  prize  for  the  victorious  concept.  Fourth,  multiple  operational  comparisons 
would  give  a  better  idea  of  the  overall  operational  capability  of  any  one  power  plant. 
This  process  outlined  in  the  steps  above  is  not  feasible  in  the  context  of  prospective 
nuclear  power  plants  because  of  the  prohibitive  cost  involved.  Other  potential  problems 
than  those  listed  above  might  also  exist. 

The  prospect  of  using  the  Mobile  Autonomous  Systems  Laboratory  for  this 
research  is  exciting  for  all  of  the  reasons  that  the  nuclear  power  plant  data  are  not.  First, 
multiple  robot  designs  with  the  same  end  requirements  were  actually  built.  Second, 
because  the  robots  were  built,  they  could  be  tested  and  compared  against  the  initial 
projections  of  success  by  each  robot  design  team.  Third,  projections  of  success  were  not 
only  made  by  the  individual  concept  designers,  but  also  by  their  peers  (i.e.,  competitors), 
allowing  for  a  much  less  biased  evaluation.  In  addition,  none  of  the  competitors  had 
anything  to  gain  monetarily  from  any  particular  evaluation  ensuring  a  less  biased 
evaluation  by  all  parties.  Fourth,  three  competitive  events  occurred  in  the  course  of  the 
robot  matches  (i.e.,  two  exhibition  matches  followed  by  the  competition)  allowing  for  a 
more  exact  assessment  of  the  overall  capability  of  any  one  robot  design. 
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Chapter  2 


Rules  of  the  robot  contest 

2.1  The  playing  field 

The  playing  field  used  in  exhibition  1  is  shown  in  Figure  1.  The  playing  fields 
used  in  exhibition  2  and  in  the  competition  were  similar  to  that  shown  in  Figure  1.  The 
robot  teams  did  not  know  the  shape  or  size  of  the  playing  field  prior  to  any  of  the  three 
events.  The  characteristics  of  the  playing  field  that  were  known  were  the  following: 
either  six  or  12  inch  tall  white  walls  with  a  blue  stripe  at  the  top,  green  floors,  red  targets, 
and  yellow  scoring  areas.  The  fact  that  the  size  and  shape  of  the  playing  field  were 
unknown  prior  to  an  event  required  that  the  robots  be  able  to  respond  dynamically  to  their 
sensor  data  [  3]. 


Figure  1:  Playing  field  for  exhibition  1  [  3] 
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2.2  How  points  were  awarded 

In  the  contests,  four  minutes  were  allowed  for  each  robot  in  the  playing  field. 
Numerous  scoring  factors  were  used.  See  Table  1  below  for  a  complete  description  of 
the  various  scoring  factors  [  3]. 


Table  1:  Methods  by  which  robots  score  points 


Action  (of  robot) 

Point(s) 

Jsll  Si  MSskljsMUki  jJSSHlHi 

1 

Transfers  target  to  yellow  scoring  area 

3 

Transfers  target  to  home 

5 

Returns  to  home  prior  to  end  of  round 

1 

Moves  after  four  minutes 

-1 

The  first  points  that  were  awarded  in  all  cases  were  for  approaching  a  target,  a 
small  red  can,  and  signaling.  Signaling  meant  that  the  robot  recognized  the  target  and 
made  some  visible  display  to  that  effect.  Signaling  would  have  been  any  recognizable 
action  such  as  spinning,  playing  music,  making  a  noise  such  as  a  whistle,  raising  a  flag, 
etc.  One  point  was  awarded  for  each  target  that  was  approached  and  for  which  a  signal 
was  accomplished  [  3]. 

After  signaling,  the  robot  grasped  the  target,  whether  that  meant  physically 
attaching  the  target  to  the  robot  by  means  of  a  magnetic  arm,  grasping  with  pincers, 
dragging  as  the  robot  moved  over  a  target,  or  simply  carrying  the  target  inside  or  on  top 
of  the  robot.  Taking  the  target  to  a  yellow  scoring  area  or  to  home  was  essential  to 
scoring  the  next  set  of  points.  Transporting  the  target  to  within  eight  inches  of  a  yellow 
scoring  area  (yellow  rectangular  patch  on  the  wall  of  the  playing  field)  granted  the  robot 
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three  points  for  each  target  left  at  the  yellow  scoring  area.  Five  points,  however,  were 
awarded  if  the  robot  instead  transferred  a  target  to  home.  Home  was  the  location  used  as 
the  starting  point  for  each  of  the  competing  robots  at  the  start  of  the  four-minute  playing 
time  [3]. 

Two  final  means  existed  by  which  points  were  awarded/deducted.  These  were 
either  returning  home  prior  to  the  end  of  the  allotted  four  minutes  (for  which  one  point 
was  awarded)  or  continuing  to  move  after  the  four  minutes  have  passed  (for  which  one 
point  was  deducted  as  a  penalty)  [3]. 
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Chapter  3 


Description  of  data  obtained  via  questionnaires 

3.1  Projections  of  future  performance  made  by  participating  teams 

The  questions  were  divided  into  three  groups  called  goals.  The  three  goals  were 
labeled  as  control,  control/score,  and  score.  The  names  of  the  three  goals  are  illustrative 
of  the  content  of  the  questions  within  the  goal  grouping.  This  means  that  the  control 
group  contains  questions  regarding  robot  control;  the  control/score  group  contains 
questions,  which  demonstrate  a  combination  of  robot  control  as  well  as  scoring  ability; 
and  finally  the  score  group  includes  questions,  which  specifically  look  at  the  scoring 
capability  of  the  robots.  The  control/score  group,  also,  was  broken  into  two  subgroups 
called  category  1  and  category  2  which  illustrate  capabilities  relating  to  targets  and 
scoring  respectively. 

In  the  first  and  second  exhibition  matches,  17  questions  were  used  to  project  the 
future  success  of  each  of  the  teams  in  the  exhibitions  (Appendix  A),  The  evaluations 
provided  by  the  participating  teams  are  found  in  Appendix  B  and  C  for  Exhibitions  1  and 
2,  respectively.  The  questionnaire  for  the  competition  match  (i.e,,  the  match  following 
the  second  exhibition  match)  included  two  additional  questions  (number  4:  robot’s  arrival 
at  targets  and  numbers  18-23  which  ask  for  the  projected  success  ranking  of  each  of  the 
participating  teams).  The  questionnaire  for  the  competition  match  also  deleted  question 
number  3  from  the  1st  and  2nd  exhibition  match  questionnaire  because  none  of  the  robots 
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damaged  the  playing  surface  in  any  noticeable  way  (i.e.,  this  question  did  not 
discriminate  between  the  robots).  See  Appendix  D  for  the  Competition  questionnaire. 
See  Appendix  E  for  the  evaluations  provided  by  the  participating  teams  prior  to  the 
Competition. 

Each  team  participating  (i.e.,  six  teams  participated  and  four  teams  did  not)  in  this 
project  completed  a  questionnaire  for  itself  and  each  of  the  other  participating  teams  by 
defining  a  range  between  1  and  10  over  which  the  robot  in  question  could  be  expected  to 
perform  in  each  of  the  questions  asked.  Some  exceptions  to  this  are  the  following:  team 
2  had  questionnaires  filled  out  regarding  its  projected  success  but  did  not,  itself,  fill  out 
questionnaires;  team  6  filled  out  questionnaires  for  the  2nd  exhibition  and  the  competition 
but  did  not  complete  any  questionnaires  for  the  1st  exhibition. 

3.2  Data  tabulated  from  results  of  Exhibitions  and  Competition 

Additionally,  William  Hardman,  who  observed  each  of  the  exhibitions  and  the 
competition,  tabulated  results.  The  results  were  tabulated  by  William  Hardman’s 
completion  of  one  questionnaire  for  each  of  the  participating  teams  during  each  of  the 
three  events.  The  difference  (as  compared  to  the  questionnaires  filled  out  by  the  teams 
prior  to  the  three  events)  in  this  case  was  that  the  result  for  a  specific  question  was 
assigned  to  only  one  bin,  not  a  range  of  bins.  The  proceeding  was  accurate  because  these 
results  were  discrete  physical  outcomes,  not  projections  of  future  success.  This  means 
that  one  bin  in  each  question  was  assigned  a  value  of  unity  (i.e.,  the  one  bin  which  best 
indicated  the  success/failure  of  the  team  to  meet  the  performance  requirement  of  that 
question). 
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Each  question  was  weighted  as  previously  discussed  in  Chapter  4.  Each  of  the 
questions  formed  a  part  of  a  group  (control,  control  and  score,  and  score)  of  questions, 
with  each  group’s  total  weighting  summing  to  unity  (as  previously  discussed).  For  a 
specific  bin  of  a  given  question,  the  weighting  of  the  question  is  multiplied  by  the 
probability  of  being  in  the  bin.  For  each  bin,  this  product  is  summed  for  each  question  in 
the  group.  The  result  is  the  probability  of  being  in  any  bin  for  a  given  group.  Each  group 
was  weighted  with  total  weighting  between  the  three  groups  summing  to  unity  (as  is 
previously  discussed).  Just  as  the  questions  within  a  group  were  reduced  to 
corresponding  group  probabilities  of  being  in  specific  bins,  the  groups  were,  then, 
reduced  to  an  overall  probability  of  a  robot  team  being  in  a  specific  bin.  These 
probabilities  will  be  called  results  in  all  of  the  ensuing  discussion.  Results  for 
Exhibitions  1  and  2  and  the  Competition  are  discussed  in  Chapter  5.  Specifically,  the 
results  of  Exhibition  1,  Exhibition  2  and  the  Competition  are  shown  in  Figure  7,  Figure 
14,  and  Figure  20  respectively. 
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Chapter  4 


How  the  data  were  weighted 

4.1  Question  weighting  from  Exhibitions  1  and  2 

Weightings  (i.e.,  a  fraction  of  unity)  were  assigned  to  each  of  the  questions  within 
a  group  such  that  the  sum  of  the  weightings  for  a  group  is  equal  to  unity.  An  exception  is 
the  control/score  group  with  its  two  sub-goals  (called  categories  1  and  2),  which  each  has 
weightings  summing  to  unity.  Lastly,  each  of  the  groups  is  assigned  a  weighting  such 
that  the  sum  of  the  group  weightings  is  likewise  equal  to  unity. 

At  the  onset  of  the  project,  the  weightings  described  above  were  assigned  based 
upon  expectations  by  the  Maslab  staff  of  how  the  robot  teams  would  attempt  to  score  and 
operate,  in  general.  These  expectations  were  most  easily  quantified  in  the  score  group 
where  point  values  were  assigned  to  each  of  the  means  by  which  points  could  be  accrued 
in  the  matches.  The  point  values  assigned  to  various  activities  established  a  natural 
ranking  or  weighting  to  each  of  the  score  group  questions  (i.e.,  highest  points  correspond 
to  highest  weighting).  Points  were  assigned  as  follows:  placing  a  target  in  the  home  area 
(5  points),  placing  a  target  in  a  yellow  score  area  (3  points),  performing  a  waypoint  signal 
meaning  making  a  noise  or  mechanical  signal  upon  arriving  at  a  target  location  (1  point), 
arriving  home  after  the  end  of  the  allotted  4  minutes  (1  point),  and  moving  after  the 
allowed  4  minutes  (-1  point). 
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After  observing  the  first  and  second  exhibition  matches,  it  became  clear  that  the 
contestants  would  attempt  to  take  targets  to  the  yellow  score  areas  as  opposed  to  the 
home  area.  Therefore,  although  more  points  were  available  for  taking  a  target  home,  this 
scoring  mechanism  was  weighted  third  in  importance  (weighting  =  0.2)  after  placing 
targets  in  the  yellow  score  areas  (weighting  =  0.4)  and  making  a  signal  once  arriving  at  a 
target  or  waypoint  (weighting  =  0.3).  The  last  two  scoring  mechanisms  for  arrival  at 
home  in  less  than  four  minutes  (weighting  =  0.05)  or  moving  after  the  allotted  four 
minutes  had  expired  (weighting  =  0.05)  were  weighted  only  minimally.  The  contestants 
all  showed  themselves  capable  of  not  exceeding  the  allowed  four  minutes  operation  time; 
and,  thus,  none  would  be  negatively  affected  by  the  loss  of  one  point  for  exceeding  the 
allowed  time.  Also,  none  of  the  competitors  showed  any  indication  that  the  goal  was  to 
end  the  matches  with  the  robot  having  made  its  way  back  to  home.  Thus,  this  scoring 
mechanism  was  not  chosen  as  a  scoring  means  by  any  of  the  competitors  and  became 
non-discriminating  and  likewise  was  minimally  weighted. 

The  control/score  group  (category  1)  weightings  initially  were  equal  to  0,6  for 
arrival  at  targets,  equal  to  0.2  for  detection  of  the  targets,  and  equal  to  0.2  for  processing 
quickly  in  detecting  targets  and  score  areas.  We  assumed  that  having  high  reliability  in 
detecting  targets  and  short  processing  times  for  detecting  targets  were  essentially 
different  versions  of  the  same  question  and  thus  merited  the  same  weighting.  The  more 
quickly  a  robot  processed  the  sensor  data  and  determined  a  detection  had  occurred  made 
for  a  much  more  timely  signal  and  thus  a  more  reliable  robot.  This  was  true  as  longer 
processing  times  meant  more  time  between  when  a  robot  first  obtained  sensor  data  on  a 
target  and  when  it  determined  that  a  target  had  been  sighted  which  also  meant  more  of  a 
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delay  between  first  sensor  data  and  time  to  signal  the  detection  of  the  target.  The  robot’s 
arrival  at  the  target  was  the  most  easily  measured  (i.e.,  visually  observed)  means  by 
which  the  control  and  scoring  capability  of  the  target  can  be  measured;  and,  thus,  was 
given  the  highest  weighting. 

The  control/score  group  (category  2)  weightings  did  not  change  throughout  the 
project.  The  weightings  were  equal  to  0.4  for  getting  to  a  yellow  score  area,  equal  to  0.3 
for  detaching  the  target  from  the  robot  into  the  yellow  score  area,  equal  to  0.15  for  the 
fraction  of  the  playing  field  explored,  equal  to  0.1  for  reliability  in  detecting  the  score 
area,  and  0.05  for  attempting  to  grasp  a  target  after  failing  to  grasp  it  on  the  first  attempt. 
The  weightings  most  representative  of  success  of  the  robot  in  this  category  were  assigned 
to  the  robot’s  arrival  at  the  yellow  score  area  followed  by  the  ability  to  detach  the  target 
in  the  score  area.  This  preference  was  natural  because  the  robot  must  first  get  to  the  score 
area  else  detaching  the  target  became  irrelevant.  Percentage  of  playing  field  explored 
was  not  a  direct  link  to  success.  However,  the  idea  was  that  if  more  area  were  explored 
then  the  robot  would  be  more  successful;  this  was  only  true  if  the  area  was  explored 
efficiently  (i.e.,  long  searches  to  identify  targets,  once  near  them,  or  random  travel 
bypassing  numerous  targets  are  not  desirable).  Reliability  in  detecting  the  scoring  areas 
was  not  quite  as  easily  quantifiable  a  metric  as  actual  arrival  at  the  score  areas  and, 
though  important,  was  just  part  of  the  process  of  arrival  at  a  score  area  and  must  receive  a 
lower  weighting.  The  robots,  with  one  exception,  used  the  approach  of  dragging  targets 
wedged  in  a  bay  beneath  them,  thus  minimizing  the  importance  of  this  question  in 
discriminating  between  robots. 
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The  control  group  weightings  initially  were  equal  to  0,4  for  the  10  second 
required  computing  and  movement  delay  prior  to  the  start  of  the  4  minute  play  period 
(allowing  for  fair  start  for  all  robot  competitors),  equal  to  0.3  for  the  robot’s  ability  to 
avoid  collisions  with  large  objects  (like  walls  in  the  match  arena),  equal  to  0.2  for  having 
a  calibration  time  of  less  than  60  seconds  prior  to  a  match  (i.e.,  time  required  to  prepare 
the  robot  for  the  start  of  a  match  once  placed  in  the  home  area)  and  equal  to  0.1  for  the 
ability  to  operate  reliably  using  battery  power.  In  the  first  and  second  exhibitions  the 
robots  all  had  difficulty  especially  with  the  ten  second  delay,  collision  with  walls  and  the 
calibration  time  prior  to  the  matches.  At  that  stage  of  the  project  the  robots 
automatically  started  at  the  push  of  their  start  buttons,  collided  frequently  with  walls  and 
often  stalled  following  a  collision,  and  required  significantly  more  calibration  time  than 
the  60  seconds  allowed  prior  to  a  match.  The  weightings  described  above  indicate  the 
relative  difficulty  that  the  robots  had  in  each  performance  area.  Operating  reliably  using 
battery  power  appeared  to  be  largely  a  non-discriminator  between  the  robots  and,  thus, 
was  weighted  minimally. 
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4.2  Question  weightings  revised 


Table  2:  Initial  and  final  values  of  the  weightings  for  Control  questions 


Control  (goal) 

initial  final 

Sub-goals 

calibration  time 

0.1875 

0.025 

10  sec  required  delay 

0.3875 

0.025 

damages  playing  surface  (penalty) 

0.05 

0.01 

collision  with  objects  in  path 

0.2875 

reliable  operation  on  battery  power 

0.0875 

■EHE 

total 

1 

i 

Table  3:  Initial  and  final  values  of  the  weightings  for  Control  and  Scoring  questions 


Control  and  Scoring  (goal) 

(two  categories) 

Category  1  sub-goals 

initial 

final 

arrival  at  targets 

0.6 

reliability  in  detecting  targets 

0.2 

0.1 

long  processing  time  in  detecting  targets 

and  score  areas 

0.2 

0.3 

total 

1 

1 

Category  2  sub-goals 

Sub-goal  weighting 

final 

arriving  at  yellow  score  areas 

HHBS] 

0.4 

detaching  target  in  score  areas 

attempting  to  grasp  target  again  if  failed  on 

first  attempt 

0.05 

0.05 

reliability  in  detecting  score  areas 

■KB 

percentage  of  playing  area  explored 

l 

total 

hh 

i 
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Table  4:  Initial  and  final  values  of  the  weightings  for  Scoring  questions 


Sub-goals 

Sub-goal  weighting 

initial 

final 

robot  performs  signal  at  waypoint  (target) 

0.3125 

■EH 

robot  places  target  in  score  area 

■BE 

0.11 

robot  places  target  in  home  area 

■Emu 

0.66 

robot  returns  home  in  <  4  min,  allowed 

0.07 

total 

i 

1 

After  observing  the  final  match,  the  competition,  it  became  apparent  that  the 
weightings  had  to  be  revised  significantly  in  the  control  and  score  groups,  as  shown  in 
Table  2  and  Table  4,  respectively,  with  only  minor  revisions  being  made  in  the 
control/score  group  (category  1)  shown  in  Table  3.  These  revisions  were  made  in  order 
to  represent  more  accurately  the  importance  of  those  performance  attributes  (or  goals), 
which  reflected  best  the  results  of  the  competition.  By  the  time  of  the  competition,  the 
robots  were  in  the  best  position  to  exhibit  all  of  the  skills  developed  over  the  course  of 
the  project  as  opposed  to  only  showcasing  the  results  of  the  progressive  design 
enhancements  following  the  first  and  second  exhibitions.  Design  and  programming 
choices  that  the  winning  competitors  had  made  reflected  different  priorities  among  the  set 
of  performance  goals,  and,  thus,  dictated  different  weights  in  forecasting  accurately  the 
outcomes  of  the  competition.  This  meant  that  weights  were  adopted  such  that 
observation  of  the  attributes  contributing  most  to  the  competition  results  combined  with 
the  actual  competition  results  (i.e.,  ranking  of  competitors  at  the  conclusion  of  the 
competition)  could  provide  a  picture  closely  resembling  the  competition  outcome. 

The  most  significant  weighting  revisions  occurred  in  the  score  group  shown  in  Table 
4.  The  robots  placing  highest  in  the  competition  chose  to  take  targets  home  as  opposed  to 
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the  yellow  score  areas,  as  was  done  in  the  first  and  second  exhibitions.  This  was  the 
major  mechanism  used  by  competitors  to  score  highly  in  the  competition.  Thus,  the 
weighting  was  changed  from  0.2  to  0.65.  The  second  major  means  used  to  score  was  the 
making  of  a  signal  upon  arrival  at  a  target,  dictating  a  weighting  change  from  0.4  to  0.15. 
The  third  most  important  scoring  method  was  that  of  taking  targets  to  the  scoring  area. 
However,  this  was  largely  unused,  and,  thus,  was  weighted  accordingly  at  0.1  (previously 
at  0.3).  The  lowest  weightings  were  assigned  to  arrival  at  home  in  less  than  four  minutes 
(going  to  0.07  from  0.05)  which  was  not  a  competitor  priority  but  which  was  slightly 
more  important  than  moving  after  the  four  minutes  allowed  (0.03).  None  of  the  robots 
moved  after  the  allowed  four  minutes.  Thus,  the  question  of  performance  in  this  area 
became  a  non-discriminator  and,  therefore,  was  weighted  negligibly. 

The  control  group  weighting  revisions  largely  made  all  but  one  question  in  this 
group  irrelevant  as  discriminators.  Collision  with  large  objects  (i.e.,  walls)  became  the 
telling  question  within  this  group  because  robots  either  wasted  significant  time  or  stalled 
completely  once  a  collision  had  occurred.  Additionally,  robots  stalled  occasionally 
without  having  collided  with  anything.  These  stalls  were  accounted  for  here  also.  The 
weighting  for  collisions  and  stalls  was  assigned  a  value  of  0.8.  (0.3  previously).  Reliable 
operation  using  battery  power,  though  largely  non-discriminating,  was  discriminated 
enough  in  the  competition  to  merit  a  weighting  of  0.14  (previously  0.1).  The  questions 
regarding  the  calibration  time  requirement  and  the  ten  second  delay  prior  to  robot 
movement  and  computation  proved  to  be  non-discriminators,  and  were  both  weighted 
negligibly  at  0.03  (previously  0.2  and  0.4  respectively). 
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The  control/score  group  (category  1)  weightings  were  only  minimally  changed. 
The  arrival  at  the  targets  maintained  its  precedence  with  a  weighting  of  0.6,  but  the 
processing  time  needed  to  detect  targets  and  score  areas  was  raised  from  0.2  to  0.3  to 
account  for  the  fact  that  this  question  included  detection  of  not  only  targets  but  also  score 
areas  (a  fact  neglected  in  the  first  weighting  assessment).  Similarly,  the  weighting  for 
reliability  in  detection  of  targets  was  lowered  from  0.2  to  0.1  in  order  to  account  for  the 
incorrect  weighting  assessment  just  described. 

4.3  Group  weightings 


Table  5:  Initial  and  final  values  of  the  weightings  for  Control  and  Scoring  sub-goals 


Control  and  Scoring  (goal) 

weightings 

initial 

final 

category  1  sub-goals 

category  2  sub-goals 

0.5 

0.3 

0.5 

0.7 

Table  6:  Initial  and  final  values  of  the  weightings  for  Control,  Control  and  Scoring,  and  Scoring 
goals 


Goa!  weightings 

Control  (goal) 

0.3 

0.1 

0.2 

0.3 

Scoring  (goal) 

0.5 

0.6 

total 

1 

1 
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The  groups  were  initially  assigned  weightings  of  0.3  for  control,  0.2  for  control 
and  score  (with  0.5  assigned  to  each  of  categories  1  and  2  as  shown  in  Table  5),  and  0.5 
for  score.  In  order  to  accurately  represent  the  results  of  the  competition  the  group 
weightings  were  changed  to  0.1  for  control,  0.3  for  control  and  score  (with  0.3  for 
category  1  and  0.7  for  category  2  as  shown  in  Table  5),  and  0.6  for  score.  See  Table  6  for 
the  initial  and  final  group  weightings. 

In  the  control  group  only  one  question  appeared  to  be  a  real  discriminator,  as 
discussed  previously.  Furthermore,  this  question  was  not  a  real  descriminator  in  points 
scored  by  the  victorious  robots;  hence  a  weight  reduction  of  0.2  was  merited  in  this  group 
(from  0.3  to  0.1). 

Based  on  how  points  were  scored  by  the  robots  in  the  competition,  no  different 
(from  that  in  the  first  and  second  exhibitions)  discrimination  appeared  to  exist  for  the 
control  and  score  group  compared  to  the  score  group.  Therefore,  the  difference  in 
weightings,  as  previously  assigned,  was  maintained.  This  meant  that  the  0.2  weight 
reduction  for  the  control  group  was  assigned  equally  between  the  control  and  score  group 
and  the  score  group  (see  Table  6). 

The  control  and  score  category  1  and  2  weightings  (see  Table  5)  should  not  have 
been  equal  given  the  fact  that  more  questions  are  asked  in  category  2  than  in  category  1 
(5  versus  1)  as  well  as  the  fact  that  questions  were  asked  which  discriminated  more 
effectively  the  various  levels  of  success  displayed  by  the  robots  (compared  to  category  1 
questions). 


33 


Chapter  5 


Results  of  the  various  robot  matches 

Throughout  Chapter  5,  the  overall  ranking  will  specify  the  ranking  of  the  teams  as 
evaluated  by  a  specific  team  or  by  the  results  of  a  match  (i.e.,  exhibitions  1  or  2  or  the 
competition).  The  overall  ranking  in  all  of  these  cases  is  taken  from  the  expected  value 
of  the  bin  number  for  each  team.  The  bin  number  in  which  a  team  is  expected  to  be 
found  is  calculated  using  the  equation  below  for  expected  value,  E(y),  given  a  discrete 
number  of  bins  (i.e.,  n  =  10). 

Equation  1:  Expected  value  (i.e.,  number  of  the  bin  in  which  a  team  is  expected  to 
be  found)  immediately  follows  and  then  is  defined  below. 

Equation  1:  Expected  value  [4] 

n 

E(y>:=^  y  rPi  • 

i=  1 

The  probability,  p  i ,  for  any  team,  of  being  in  each  of  the  ten  bins,  respectively,  (n  =  10) 
has  been  assigned.  Multiplying  the  probability,  p  i ,  of  being  in  a  bin  by  the  bin  number, 
yi ,  and  then  summing  over  the  ten  products  gives  the  expected  bin  in  which  a  team 
should  be  found.  The  higher  the  bin  number,  the  better  the  team  is  expected  to  perform, 
and  the  higher  ranking  that  is  achieved. 

Also,  throughout  Chapter  5,  the  word  ranking  means  a  ranking  of  highest  to 
lowest  probabilities  of  a  team  being  in  a  specific  bin.  Therefore,  this  ranking  is  also  a 
ranking  (1st  to  last)  of  teams’  performance  for  that  specific  bin. 
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Table  7:  Far  left  column  indicates  the  team  whose  evaluations  produced  the  Exhibition  1,  Exhibition 
2  and  Competition  rankings  as  a  function  of  expected  bin  in  which  a  team  should  be  found.  The 
teams  are  ranked  highest  to  lowest  with  the  highest  at  the  top  and  lowest  at  the  bottom.  To  the  right 
of  the  Exhibition  1,  Exhibition  2  and  Competition  rankings  are  the  results,  in  the  form  of  team 
rankings  also,  for  each  of  the  respective  events. 


Team  1 


Exhibition  1 


team 


2 


5 


10 


3 


6 


1 


Results 


Exhibition  1 


erformance  for 


all  teams 


Exhibition  2 


team 


Results 


Exhibition  2 


team 


10 


3 


2 


6 


1,5 


Results 

team 

team 

5 

2 

10 

10 

3 

1 

6 

6 

2 

3 

1 

5 

Team  3 


Team  5 


erformance  for 


all  teams 


SEHI 


erformance  for 


all  teams 


Team  6 


erformance  for 


all  teams 


6  10 


Team  6  eval  luations  used  Ifrom  Exhibition  2 


team 


6 


team 

team 

10 

2 

1 

10 

3 

1 

6 

6 

2 

3 

9ESS 

5 
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5.1  Exhibition  1 


5.1.1  Exhibition  1  projections  (of  success)  made  by  participating  teams 

In  Figure  2  (on  the  next  page)  see  the  evaluation  by  team  1  of  the  probability  of 
success  for  the  teams  in  exhibition  1.  Team  1  predicted  that  the  overall  ranking,  from 
first  to  last,  would  be  team  2,  team  5,  team  10,  team  3,  team  6,  and  finally  team  1  (see 
Table  7).  Given  the  minor  variation  in  probabilities  for  all  teams  in  either  bin  1  or  bin  10, 
it  is  apparent  that  team  1  believed  that  all  teams  had  an  approximately  equal  likelihood  of 
being  in  bin  1.  Likewise,  team  1  believed  all  teams  had  an  approximately  equal 
likelihood  of  being  in  bin  10. 

Team  1  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
2,  team  3,  team  5,  team  6,  team  10,  and  finally  team  1.  For  bin  number  5,  team  1 
predicted  the  ranking  (again  first  to  last)  as  team  10,  team  1,  team  6,  team  3,  team  5,  and 
finally  team  2.  In  bin  number  1,  team  1  predicted  the  ranking  (also  first  to  last)  as  team  6, 
team  1,  team  5,  team  10,  team  2,  and  lastly  team  3. 
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Probability  of  being  in  the  bin 


bin  number 

Figure  2:  Prior  evaluation  by  team  1  of  the  probability  of  success  for  the  teams  in  exhibition  1. 
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In  Figure  3  below  see  the  evaluation  by  team  3  of  the  probability  of  success  for 
the  teams  in  exhibition  1.  Team  3  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  10,  team  6,  team  2,  team  1,  team  3,  and  finally  team  5(see  Table  7).  . 
Figure  3  shows  that  teams  10, 6, 2,  and  1  were  expected  to  do  well  but  that  teams  3  and  5 
were  expected  to  do  poorly. 

Team  3  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
2,  team  10,  team  6,  team  1,  team  3,  and  finally  team  5.  For  bin  number  5,  team  3 
predicted  the  ranking  (again  first  to  last)  as  team  1,  team  2,  team  10,  team  6,  team  5,  and 
finally  team  3.  Lastly,  in  bin  number  1,  team  3  predicted  the  ranking  (also  first  to  last)  as 
team  5,  team  3,  team  1,  and  teams  2, 6,  and  10. 


— ♦ — Teeml 
»  Teem2 
-~^v»Team3 
— ^ — TeemS 
— * — Team6 


Figure  3:  Prior  evaluation  by  team  3  of  the  probability  of  success  for  the  teams  in  exhibition  1, 
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In  Figure  4  below  see  the  evaluation  by  team  5  of  the  probability  of  success  for 
the  teams  in  exhibition  1.  Team  5  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  3,  team  2,  team  5,  team  6,  team  10,  and  finally  team  l(see  Table  7).  .  As 
shown  in  Figure  4,  Team  5  expected  largely  poor  performance  from  all  but  team  3. 

Team  5  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
2,  team  5,  team  1,  team  3,  team  10,  and  finally  team  6.  For  bin  number  5,  team  5 
predicted  the  ranking  (again  first  to  last)  as  team  3,  team  10,  teams  2  and  5,  team  6,  and 
finally  team  1.  In  bin  number  1,  team  5  predicted  the  ranking  (also  first  to  last)  as  team  1, 
team  6,  teams  5  and  10,  team  2,  and  lastly  team  3. 


Figure  4:  Prior  evaluation  by  team  5  of  the  probability  of  success  for  the  teams  in  exhibition  1. 


Team  6  provided  no  evaluations  for  Exhibition  1.  Instead  of  disregarding  Team  6 
evaluations  for  Exhibition  2  and  the  Competition,  the  Team  6  evaluations  for  Exhibition 
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2  were  substituted  for  the,  non-existent,  evaluations  for  Exhibition  1.  This  fact  minimizes 
the  value  of  team  6  evaluations. 

In  Figure  5  below  see  the  evaluation  by  team  10  of  the  probability  of  success  for 
the  teams  in  exhibition  1.  Team  10  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  6,  team  2,  team  3,  team  10,  team  5,  and  finally  team  l(see  Table  7).  .  As 
shown  in  Figure  5,  team  10  expected  all  teams  to  have  moderate  success  as  shown  by  the 
approximately  equal  probabilities  (0.57  to  0.6)  of  being  in  bin  5. 

Team  10  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be 
team  3,  team  6,  team  2,  team  10,  and  finally  teams  1  and  5.  For  bin  number  5,  team  10 
predicted  the  ranking  (again  first  to  last)  as  team  10,  teams  3  and  5,  team  2,  team  1,  and 
finally  team  6.  In  bin  number  1,  team  10  predicted  the  ranking  (also  first  to  last)  as  team 
10,  team  5,  team  1,  and  lastly  teams  2,3  and  6. 


—4 — Team  t 
— ■ — Team  2 

....  g  . Team  3 

— K— -Team  5 
— ■  -'Team  6 
— 4— -Team  t> 


Figure  5:  Prior  evaluation  by  team  10  of  the  probability  of  success  for  the  teams  in  exhibition  1. 
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5.1.2  Results  of  exhibition  1 


The  only  importance  assigned  to  Figure  6  below  is  the  fact  that  none  of  the 
participating  teams  was  prepared  to  have  its  robots  take  part  in  exhibition  1.  This  means 
that  scores  of  one  (i.e.,  the  lowest  score)  were  officially  assigned  to  each  of  the 
participating  teams.  Each  of  the  participating  teams  was,  however  very  close  to  being 
prepared  for  exhibition  1,  a  fact  that  is  useful  in  the  description  for  Figure  7.  Note  that 
although  it  appears  in  Figure  6  that  only  team  2  participated  in  exhibition  1  the  one  line 
of  the  plot  applies  to  all  6  participating  teams. 


Figure  6:  Exhibition  1  results  (participating  teams).  The  one  line  of  the  plot  applies  to  all  6 
participating  teams  and  merely  indicates  that  the  participating  teams  were  not  prepared  for 
Exhibition  1. 
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Note  in  Figure  7  below  the  results  for  exhibition  1  from  the  teams  not 
participating  in  our  project  are  plotted.  This  is  done  because  none  of  the  participating 
teams  have  scores  that  are  useful  for  the  Bayesian  approach  to  projecting  the  success  of 
the  robots  in  successive  events,  which  is  discussed  in  Chapter  6.  Therefore,  an  initial 
result  for  all  of  the  participating  teams  was  taken  as  the  average  (in  each  bin)  of  the  non¬ 
participating  team  results  such  that  a  useful  starting  point  for  further  projections  could  be 
had.  These  bin-wise  averages  were  taken  to  be  the  evidence  obtained  from  tests  of  the 
likely  performance  of  a  typical  team  that  was  able  to  get  its  robot  working.  Use  of  this 
evidence  permits  Bayesian  projections  of  team-specific  performance  for  exhibition  2  and 
for  the  competition  evaluations  to  be  closer  to  reality  than  if  scores  of  unity  in  bin  #  1,  as 
shown  in  Figure  6,  were  assigned  to  each  of  the  participating  teams  for  exhibition  1. 


Figure  7:  Exhibition  1  results  (non-participating  teams) 
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Figure  8  below  graphically  illustrates  the  assumed  values  of  the  results  for  each  of 
the  participating  teams.  These  values  (bin- wise  averages  of  the  non-participating  teams 
results  for  Exhibition  1,  see  Figure  7)  were  assumed  since  the  participating  teams  were 
not  prepared  for  Exhibition  1 . 


Figure  8:  Assumed  Exhibition  1  results  for  each  of  the  participating  teams.  These  results  are  the  bin- 
wise  averages  of  the  data  presented  in  Figure  7. 
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5.1.3  A  general  description  of  faults  of  the  robots  in  Exhibition  1  follows. 

Manual  starts  of  robots  were  required.  A  manual  start  means,  simply,  that  a 
button  is  pushed  and  the  robot  begins  computing  and/or  moving  immediately  after  the 
start  button  is  depressed  (i.e.,  10-second  delay  not  met  as  discussed  below). 

Most  robots  began  moving  or  computing  prior  to  the  required  10-second  delay  at 
the  beginning  of  the  event.  The  10-second  delay  is  meant  to  ensure  that  the  robots  would 
begin  the  event  at  exactly  the  same  time,  demonstrating  control  of  the  robot. 

The  robots  spent  considerable  time  scanning  for  targets.  Once  a  target  was 
approached  many  robots  were  still  unable  to  signal  by  a  noise  or  mechanical  action  that 
the  robot  is  within  four  inches  of  the  target. 

Most  sensors  (e.g.,  ultrasound  or  infrared  range  finders)  were  not  operational.  For 
most  robots  vision  was  the  only  sensor  as  of  yet  at  their  disposal.  This  made  it  possible 
to  detect  the  targets  but  impossible  to  stop  (because  only  the  code  for  target  detection  and 
movement  toward  the  detected  target,  not  what  to  do  once  target  was  approached,  was 
written  at  this  point)  and  signal;  hence  robots  just  drove  through  the  targets  without 
stopping.  Once  the  robot  drove  over  the  target  it  lost  sight  of  the  target  and  began 
looking  for  other  targets. 


5.2  Exhibition  2 

5.2.1  Exhibition  2  projections  (of  success)  made  by  participating  teams 

In  Figure  9  (next  page)  see  the  evaluation  by  team  1  of  the  probability  of  success 
for  the  teams  in  exhibition  2.  Team  1  predicted  that  the  overall  ranking,  from  first  to  last, 
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would  be  team  1,  team  3,  team  6,  team  10,  team  5,  and  finally  team  2  (see  Table  7).  This 
prediction  was  accurate  for  two  teams,  teams  3  and  5.  Teams  5  and  2  were  expected  to 
perform  poorly;  teams  6  and  10  were  expected  to  be  about  average  performers;  while 
teams  1  and  3  were  expected  to  be  good  performers. 

Team  1  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
1,  team  3,  team  10,  team  5,  team  6,  and  finally  team  2.  For  bin  number  5,  team  1 
predicted  the  ranking  (again  first  to  last)  as  team  2,  team  6,  team  1,  team  10,  team  3,  and 
finally  team  5.  In  bin  number  1,  team  1  predicted  the  ranking  (also  first  to  last)  as  team 
10,  team  3,  team  5,  team  6,  team  2,  and  lastly  team  1. 


♦  Team  1 
— ■ — Team  2 
-  i:-  Team  3 
— H — Team  5 
— 3K — Team  6 
— • — Team  10 


Figure  9:  Evaluation  by  team  1  of  the  probability  of  success  for  the  teams  in  exhibition  2. 
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In  Figure  10  below  see  the  evaluation  by  team  3  of  the  probability  of  success  for 
the  teams  in  exhibition  2.  Team  3  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  10,  team  2,  team  6,  team  1,  team  5,  and  finally  team  3(see  Table  7).  Team 
3  was  accurate  on  two  teams  (teams  10  and  5).  Team  3  expects  poor  performance  from 
team  3;  average  performance  from  teams  1, 5,  and  6;  and  good  performance  from  teams  2 
and  10. 

Team  3  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
10,  team  2,  teams  1,5  and  6,  and  finally  team  3.  For  bin  number  5,  team  3  predicted  the 
ranking  (again  first  to  last)  as  team  6,  team  5,  team  1,  team  3,  team  2,  and  finally  team  10. 
In  bin  number  1,  team  3  predicted  the  ranking  (also  first  to  last)  as  team  3,  team  1,  and 
teams  2,  5, 6,  and  10. 


■  4— Team  1 
— 9 — Team  2 

. #  Team  3 

— -K — Team  5 
— 3K — Team  6 
— • — Team  10 


bin  number 


Figure  10;  Evaluation  by  team  3  of  the  probability  of  success  for  the  teams  in  exhibition  2. 
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In  Figure  1 1  below  see  the  evaluation  by  team  5  of  the  probability  of  success  for 
the  teams  in  exhibition  2.  Team  5  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  3,  team  2,  team  1,  team  10,  team  6,  and  finally  team  5(see  Table  7).  This 
prediction  was  accurate  only  for  team  5.  Team  5  predicted  poor  performance  from  team 
5;  average  performance  from  teams  1,  2,  6,  and  10;  and  good  performance  from  team  3. 

Team  5  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
3,  team  5,  team  2,  team  1,  team  6,  and  finally  team  10.  For  bin  number  5,  team  5 
predicted  the  ranking  (again  first  to  last)  as  team  1,  team  2,  team  10,  team  6,  team  3,  and 
finally  team  5.  In  bin  number  1,  team  5  predicted  the  ranking  (also  first  to  last)  as  team  5, 
team  6,  team  10,  and  teams  1,2,  and  3. 


— ♦ — Teaml 
— » — Team2 
Tean3 
' Teems 
— * — Teem6 
"  •  Teant) 


Figure  11:  Evaluation  by  team  5  of  the  probability  of  success  for  the  teams  in  exhibition  2. 
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In  Figure  12  below  see  the  evaluation  by  team  6  of  the  probability  of  success  for 
the  teams  in  exhibition  2.  Team  6  predicted  that  the  overall  ranking,  from  first  to  tot, 
would  be  team  5,  team  1,  team  3,  team  2,  team  6,  and  finally  team  10  (see  Table  7). 
Team  6  was  accurate  on  none  of  these  predictions.  Team  6  predictions  were  somewhat 
inconsistent:  team  10  poor  or  good;  team  5  poor,  average  or  good;  team  3  poor  or  good; 
teams  2  and  6  about  average,  team  1  good. 

Team  6  predicted  the  ranking  (i.e„  first  to  tot)  for  bin  number  10  would  be  team 
5,  team  10,  team  1,  team  3,  team  6,  and  finally  team  2.  For  bin  number  5,  team  6 
predicted  the  ranking  (again  first  to  last)  as  team  1,  team  6,  team  2,  team  5,  team  3,  and 
finally  team  10.  In  bin  number  1,  team  6  predicted  the  ranking  (also  first  to  last)  as  team 
10,  team  5,  team  3,  team  2,  team  6,  and  lastly  team  1. 


♦  Team  1 
— • — Team  2 
Team  3 
— H~~Team  5 
— 9K — Team  6 
— •—Team  10 


Figure  12;  Evaluation  by  team  6  of  the  probability  of  success  for  the  teams  In  exhibition  2, 
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In  Figure  13  below  see  the  evaluation  by  team  10  of  the  probability  of  success  for 
the  teams  in  exhibition  2.  Team  10  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  2,  team  10,  team  3,  team  5,  team  1,  and  finally  team  6  (see  Table  7). 

Team  10  was  accurate  on  one  team  (team  1).  Team  10  predicted  poor  performance  from 
teams  1, 3,  5,  6  and  10  and  good  performance  from  team  2. 

Team  10  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be 
team  10,  team  3,  team  5,  team  2,  team  1,  and  finally  team  6.  For  bin  number  5,  team  10 
predicted  the  ranking  (again  first  to  last)  as  team  2,  team  6,  team  5,  team  1,  team  10,  and 
finally  team  3.  In  bin  number  1,  team  10  predicted  the  ranking  (also  first  to  last)  as  team 
1,  team  6,  team  3,  team  5,  team  10,  and  lastly  team  2. 


■  "■  ♦  Teaml 
— ■ — Team  2 
Team  3 
— Wr — Team  5 
— 3K — Team  6 
♦  Team  10 


Figure  13:  Evaluation  by  team  10  of  the  probability  of  success  for  the  teams  in  exhibition  2. 
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5.2.2  Results  of  exhibition  2 


In  Figure  14  below  see  the  results  for  the  teams  in  exhibition  2.  The  overall 
ranking,  from  first  to  last,  was  team  10,  team  3,  team  2,  team  6,  and  teams  1  and  5  (see 
Table  7).  The  results  show  poor  performance  for  teams  1, 2, 6,  and  5;  slightly  better  than 
poor  performance  for  team  3;  and  still  slightly  better  performance  from  team  10. 
Summarizing,  poor  results  were  seen  from  all  teams. 

The  results  ranking  (i.e.,  first  to  last)  for  bin  number  10  was  teams  2  and  10, 
teams  3  and  6,  and  teams  1  and  5.  For  bin  number  5,  the  ranking  (again  first  to  last)  was 
team  3,  team  2,  team  6,  and  teams  1, 5,  and  10.  In  bin  number  1,  the  ranking  (also  first  to 
last)  was  teams  1  and  5,  team  6,  team  2,  team  3,  and  lastly  team  10. 


Figure  14:  Exhibition  2  results  (participating  teams) 
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5.2.3  A  general  description  of  faults  of  the  robots  in  Exhibition  2  follows.  The 
robots  in  Exhibition  2  were  still  ramping  up  to  attaining  the  final  capabilities  that 
will  be  exhibited  in  the  Competition. 

The  robots  were  not  moving  for  a  long  time  after  start  of  the  four-minute  contest 
period.  Worse,  the  robots  often  moved  only  minimally  from  the  home  position  (i.e., 
starting  point)  and  then  stalled  requiring  a  restart  of  the  robot’s  computer. 

The  robots  still  spent  considerable  time  scanning  for  targets.  Upon  arrival  at  a 
target,  some  robots  were  still  not  signaling.  Additionally,  some  robots  did  not  pick  up 
the  targets  but,  rather,  spun  them  endlessly  instead.  What  targets  were  picked  up  were 
not  readily  detached  in  scoring  areas. 
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5.3  The  competition 

5.3.1  Competition  projections  (of  success)  made  by  participating  teams 

In  Figure  15  below  see  the  evaluation  by  team  1  of  the  probability  of  success  for 
the  teams  in  the  competition.  Team  1  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  5,  team  10,  team  3,  team  6,  team  2,  and  finally  team  1  (see  Table  7). 

Team  1  was  accurate  on  two  teams  (teams  6  and  10).  Team  1  predicted  poor 
performance  from  teams  1  and  2  and  good  performance  from  teams  1, 5, 6  and  10. 

Team  1  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
10,  team  3,  team  6,  team  1,  team  5,  and  finally  team  2.  For  bin  number  5,  team  1 
predicted  the  ranking  (again  first  to  last)  as  team  2,  team  5,  team  3,  team  6,  team  10,  and 
finally  team  1.  In  bin  number  1,  team  1  predicted  the  ranking  (also  first  to  last)  as  team  1, 
team  2,  team  10,  team  6,  team  3,  and  lastly  team  5. 


— ♦— Team  1 
— * — Team  2 
— >  — Team  3 
—X—Team  5 
— — Team  8 
— • — Team  10 


Figure  15:  Evaluation  by  team  1  of  the  probability  of  success  for  teams  in  the  competition. 
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In  Figure  16  below  see  the  evaluation  by  team  3  of  the  probability  of  success  for 
the  teams  in  the  competition.  Team  3  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  3,  team  10,  team  2,  team  6,  team  1,  and  finally  team  5  (see  Table  7). 

Team  3  was  accurate  on  two  teams  (6  and  10).  Team  3  expected  poor  performance  from 
team  5;  below  average  performance  from  teams  1,  2,  and  6;  average  performance  from 
team  10;  and  good  performance  from  team  3. 

Team  3  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
3,  team  10,  team  2,  teams  1  and  5,  and  finally  team  6.  For  bin  number  5,  team  3 
predicted  the  ranking  (again  first  to  last)  as  team  6,  team  1,  team  2,  team  3,  team  10,  and 
finally  team  5.  In  bin  number  1,  team  3  predicted  the  ranking  (also  first  to  last)  as  team  5, 
team  2,  and  teams  1,  3,  6,  and  10. 


♦  Team  1 
— » — Team  2 
Team  3 
— Team  5 
— * — Team  6 
— • — Team  10 


Figure  16:  Evaluation  by  team  3  of  the  probability  of  success  for  teams  in  the  competition. 
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In  Figure  17  below  see  the  evaluation  by  team  5  of  the  probability  of  success  for 
the  teams  in  the  competition.  Team  5  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  10,  team  1,  team  3,  team  6,  team  2,  and  finally  team  5  (see  Table  7). 

Team  5  was  accurate  on  one  team  (team  6),  Team  5  predictions  were  somewhat 
inconsistent:  team  1  poor  or  good,  team  6  poor  or  good,  team  2  poor,  team  5  poor,  team  3 
average,  and  team  10  good. 

Team  5  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
10,  team  6,  team  1,  team  3,  team  2,  and  finally  team  5.  For  bin  number  5,  team  5 
predicted  the  ranking  (again  first  to  last)  as  team  3,  team  6,  team  5,  team  2,  team  1,  and 
finally  team  10.  In  bin  number  1,  team  5  predicted  the  ranking  (also  first  to  last)  as  team 
6,  team  2,  team  5,  team  1,  team  3,  and  lastly  team  10. 


. -Team  1 

— ■—Team  2 
— Team  3 
— *f~Team  5 
X  Team  6 
— # — Team  10 


Figure  17:  Evaluation  by  team  5  of  the  probability  of  success  for  teams  in  the  competition. 


54 


In  Figure  18  below  see  the  evaluation  by  team  6  of  the  probability  of  success  for 
the  teams  in  the  competition.  Team  6  predicted  that  the  overall  ranking,  from  first  to  last, 
would  be  team  10,  team  1,  team  3,  team  6,  team  2,  and  finally  team  5  (see  Table  7). 

Team  6  was  accurate  for  two  teams  (teams  5  and  6).  Team  6  predictions  were  somewhat 
inconsistent:  team  2  poor  or  good,  team  6  poor  or  good,  team  10  poor  or  good,  team  5 
poor,  team  3  average,  and  team  1  average. 

Team  6  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be  team 
6,  team  10,  team  2,  team  3,  team  1,  and  finally  team  5.  For  bin  number  5,  team  6 
predicted  the  ranking  (again  first  to  last)  as  team  1,  team  6,  team  5,  team  3,  team  2,  and 
finally  team  10.  In  bin  number  1,  team  6  predicted  the  ranking  (also  first  to  last)  as  team 
6,  team  10,  team  5,  team  2,  team  3,  and  lastly  team  1. 


bin  number 


■Team  1 
Team  2 
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Team  5 
Team  6 
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Figure  18:  Evaluation  by  team  6  of  the  probability  of  success  for  teams  in  the  competition. 
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In  Figure  19  below  see  the  evaluation  by  team  10  of  the  probability  of  success  for 
the  teams  in  the  competition.  Team  10  predicted  that  the  overall  ranking,  from  first  to 
last,  would  be  team  10,  team  2,  team  3,  team  6,  team  5,  and  finally  team  1  (see  Table  7). 
Team  10  was  accurate  on  one  team  (team  6).  Team  10  predicted  poor  performance  for 
team  1,  poor  performance  for  team  5,  average  performance  for  team  3,  average 
performance  for  team  6,  good  performance  for  team  10,  and  good  performance  by  team 
2. 

Team  10  predicted  the  ranking  (i.e.,  first  to  last)  for  bin  number  10  would  be 
team  10,  team  2,  team  3,  and  teams  1, 5,  and  6.  For  bin  number  5,  team  10  predicted  the 
ranking  (again  first  to  last)  as  team  6,  team  3,  team  1,  team  2,  team  5,  and  finally  team  10, 
In  bin  number  1,  team  10  predicted  the  ranking  (also  first  to  last)  as  team  1,  team  10, 
team  5,  team  2,  team  3,  and  lastly  team  6. 


♦  Team  1 
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Figure  19:  Evaluation  by  team  10  of  the  probability  of  success  for  teams  in  the  competition. 
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5.3.2  Results  of  competition 

In  Figure  20  (shown  on  the  next  page)  see  the  results  for  the  teams  in  the 
competition.  The  overall  ranking,  from  first  to  last,  was  team  2,  team  10,  team  1,  team  6, 
team  3,  and  finally  team  5.  The  results  indicate  that  teams  3,  5  and  6  performed  poorly, 
that  team  1  was  an  average  performer,  and  that  teams  2  and  10  were  good-to-excellent 
performers. 

The  ranking  (i.e.,  first  to  last)  for  bin  number  10  was  team  2,  team  10,  team  1, 
team  5,  and  teams  3  and  6.  Note  in  Table  8  (shown  on  the  next  page)  that  the  actual 
ranking  by  points  scored  in  the  Competition  is  shown  as  team  10,  team  2,  team  1,  teams  5 
and  6,  and  team  3  indicating  some  error  in  the  weighting  scheme  (discussed  in  chapter  4) 
used  to  predict  outcomes  of  events  as  well  as  display  results  of  events.  For  bin  number  5, 
the  ranking  (again  first  to  last)  was  team  10,  team  1,  teams  3  and  6,  team  2,  and  team  5. 

In  bin  number  1,  the  ranking  (also  first  to  last)  was  team  5,  team  3,  team  6,  team  2,  and 
teams  1  and  10. 
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Figure  20:  Competition  results  (participating  teams). 


Table  8:  Competition  results  in  terms  of  points  awarded. 


Team  number 
10 
2 
1 

5,6 

3 


Points  scored 
12 
11 
4 

1 

0 


58 


5.3.3  A  general  description  of  faults  of  the  robots  in  the  Competition  follows. 

Some  robots  still  exhibited  difficulty  in  signaling  the  arrival  at  a  target.  Others, 
however,  readily  lifted  a  mechanical  arm,  played  some  music  or  made  a  whistle. 

False  signals  occurred  (i.e.,  signaling  without  being  near  a  target). 

Some  robots  stalled  in  a  comer,  at  a  wall,  at  scoring  areas  or  at  home.  Some 
robots  even  stalled  without  having  left  home. 

Numerous  difficulties  were  exhibited  in  dealing  with  targets  either  in  aspects  of 
detection  or  of  control.  Some  robots  drove  past  or  over  targets  or  past  score  areas.  Other 
robots  pushed  targets  to  score  areas  only  to  then  push  them  out  of  those  same  score  areas. 
Still  other  robots  were  able  to  collect  numerous  targets  but  were  then  unable  to  transport 
them  to  score  areas. 
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Chapter  6 


Bayesian  approach  to  projecting  success  of  robots  in  successive 
events 

The  value  of  Bayes’  Theorem  lies  in  the  ability  to  predict  the  success  of  future 
events,  in  our  case  robot  performance,  in  successive  events.  The  requirements  to  use 
Bayes’  Theorem  are  an  initial  data  set  predicting  the  success  of  the  robots,  in  our  case, 
and  the  results  of  that  event  (i.e,,  evidence).  The  participating  teams  provided  the 
predictions  of  success  for  each  of  the  participating  teams  by  way  of  the  questionnaires 
(i.e.,  evaluations  projecting  the  success  of  the  teams  in  Exhibition  1).  The  results  of 
Exhibition  1  (and  the  subsequent  events)  were  compiled  by  William  Hardman  and  are 
described  in  detail  in  Section  3.2. 

The  two  data  sets  just  described  (predictions  of  Exhibition  1  and  the  results  of 
Exhibition  1)  can  then  be  combined,  using  Bayes’  Theorem  in  producing  the  Bayesian- 
updated  prediction  for  the  next  event  (Exhibition  2).  This  Bayesian-updated  prediction 
for  Exhibition  2  can  then  be  combined  with  the  results,  or  evidence,  from  Exhibition  2, 
by  using  Bayes’  Theorem  for  the  second  time  to  produce  the  Bayesian-updated  prediction 
for  the  next  event  (i.e.,  the  competition).  The  point  is  simple;  the  output  of  the  first  use  of 
Bayes’  Theorem  provides  input  for  the  subsequent  use  of  the  theorem  provided  new 
evidence  exists.  If  new  evidence  exists,  then  the  second  use  of  Bayes5  Theorem  can 
occur  and  provide  a  prediction  of  the  next  event.  This  process  can  be  repeated  as  long  as 
additional  events  occur  and  provide  new  evidence,  each  time. 
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6.1  Description  of  the  Bayesian  method 


Bayes’  Theorem  (equation  2)  is  that  which  immediately  follows  with  individual 
parts  of  the  equations  defined  subsequently. 

Equation  2:  Bayes’  Theorem  [  4] 


p(e  = 

p(s,e  =  e1) 

P(e  =  *0 

p(e  =  ^s) 


p(s,e  =  ^)p(0  =  e,) 

n 

2)  p(«-e  =  *.)p(e  =  »i) 
i=  l 

i  s  the  li  keli  hood  of  experi  mental  outcome  s  if 
6  =  d1  (conditional  probability) 

the  prior  probability  of  0  =  61  that  is  prior  to  the  ei 
availability  of  the  experimental  information  s 

the  posterior  probability  of  0  =  et  that  is,  the  probability 
that  has  been  revised  in  light  of  experimental  outcome  s 


Equation  2,  or  Bayes’  Theorem,  allows  for  consistent  revision  of  performance 
expectations  as  new  evidence  becomes  available  that  is  relevant  to  the  success  of  the 
robots  in  subsequent  events.  The  prior  probability  described  above  is  the  evaluating  (i.e., 
our  participating)  team’s  predictions  of  success  (obtained  from  the  questionnaires)  for 
each  of  the  participating  teams  for  Exhibition  1.  Given  the  prior  probability  and  the 
results  of  the  robot  teams  in  Exhibition  1  (i.e.,  the  actual  evidence  of  the  robots’ 
performance),  a  Bayesian-updated  prediction  of  success  (i.e.,  the  posterior  probability) 
for  Exhibition  2  is  obtained  using  equation  1.  Of  note  here  is  the  fact  that  the  average  of 
the  four  non-participating  teams  results  was  used  as  the  results,  or  evidence,  for  each  of 
the  participating  teams  as  the  participating  teams  were  of  similar  capability  though  not 
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quite  prepared  to  participate  in  Exhibition  1.  Note  also,  that  the  results  of  the  robots5 
performance,  or  evidence,  correlates  with  the  conditional  probability  in  Bayes’  Theorem. 
The  above  means  that  a  posterior  probability  is  obtained  for  each  distribution  obtained 
from  the  other  teams.  Of  course,  this  means  that  the  sum  of  the  probabilities  of  all  ten 
bins,  for  any  one  team,  remains  equal  to  unity. 

Bayes’  Theorem  is  used  a  second  time  to  obtain  a  new  posterior  distribution  based 
upon  the  team-specific  results  of  Exhibition  2.  The  new  posterior  distribution  is  an 
indication  of  the  projected  success  of  the  robots  in  the  Competition  (i.e.,  the  next  event). 
This  requires  that  the  posterior  probability  (the  Bayesian-updated  prediction  of  success 
for  Exhibition  2)  obtained  above  becomes  the  prior  probability  for  this  next  use  of  Bayes’ 
Theorem.  That  is,  to  obtain  the  current  Bayesian-updated  prediction  of  success  in  future 
events  (i.e.,  the  competition),  the  calculated  probability  (Bayesian-updated  prediction  of 
success  for  Exhibition  2)  of  success  for  the  previous  event  must  itself  be  updated,  which 
is  one  way  of  looking  at  what  Bayes’  Theorem  accomplishes. 

Therefore,  in  using  Bayes’  Theorem  a  second  time  we  take  the  new  prior 
probability  just  described  in  conjunction  with  the  results  from  Exhibition  2  (i.e,,  the 
evidence  of  the  robots’  performance)  to  obtain  the  posterior  probability,  applicable  before 
the  Competition.  The  posterior  probability  relevant  to  the  Competition  is  the  Bayesian- 
updated  prediction  of  success  for  the  Competition  obtained  by  the  same  method  as  the 
posterior  probability  for  Exhibition  2. 
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6.2  Error  Calculation 


Once  Bayesian-updated  predictions  of  success  have  been  calculated  for 
Exhibition  2  and  the  Competition,  respectively,  the  next  step  in  our  analysis  is  to 
calculate  root  mean  square  errors  (RMS)  concerning  these  predictions  of  success  and  the 
actual  results  of  the  two  events.  The  root  mean  squared  error  is  obtained  as  shown  in 
Equation  3.  below. 


Equation  3:  Root  Mean  Squared  Error  (RMS)  [  5] 


RMS  = 


In  this  analysis  we  take  the  sum  of  the  difference  squared  between  the  actual 

results  (  yi  above)  and  the  predicted  results,  yi,  then  divide  by  the  total  number  of  terms, 
n  (i.e.,  10,  corresponding  to  10  bins),  and  finally  take  the  square  root  in  order  to  obtain 
the  RMS. 

Of  value  are  two  calculations  of  the  RMS:  that  of  the  Bayesian  predicted  results 
and  actual  results  and  that  of  the  team  projections  of  success  and  actual  results  (both 
calculations  done  for  Exhibition  2  and  the  Competition).  These  calculations  can  then  be 
compared  to  those  in  the  paragraph  above  to  determine  whether  Bayesian  updating  more 
closely  approximated  the  actual  results  or  if  the  teams  were  able  to  predict  more 
effectively  the  actual  results.  The  method  with  the  lower  RMS  indicates  that  that  method 
has,  in  fact,  predicted  more  effectively  the  actual  results. 
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6.2.1  RMS  error  for  Exhibition  2 

The  upper  half  of  Table  9  and  Figure  21  (on  the  next  page)  provide  a  tabulation  of 
average  RMS  error  and  a  graphical  representation  of  RMS  error  between  actual  results 
and  Bayesian  projections  for  Exhibition  2,  respectively.  The  tabulated  data  in  Table  9  are 
a  sum  of  the  RMS  errors  as  well  as  the  average  RMS  error  for  all  of  the  teams  evaluated 
by  teams  1, 3, 5, 6  and  10  (i.e.,  teams  1, 3, 5, 6  and  10  each  evaluated  teams  1, 2, 3, 5, 6 
and  10).  An  RMS  error  was  determined  for  each  of  the  6  teams  evaluated;  this  was  done 
for  each  of  the  5  evaluating  teams  (i.e.,  a  total  of  30  calculations).  An  average  RMS 
error,  for  each  evaluating  team,  was  obtained  by  taking  a  simple  average  of  the  RMS 
error  values  (6  total  values)  between  a  teams  performance  in  the  event  (Exhibition  2)  and 
the  Bayesian-updated  prediction  of  that  team’s  success.  Figure  21  offers  the  graphical 
representation  of  the  data,  which  are  reduced  to  average  RMS  error  in  the  upper  half  of 
Table  9.  Therefore,  the  evaluating  team  (for  Exhibition  2)  with  the  lowest  average  RMS 
error  was  the  team  whose  Bayesian-updated  predictions  for  Exhibition  2  most  accurately 
predicted  the  results  of  exhibition  2.  For  Exhibition  2,  a  ranking  of  the  teams  was 
determined  (lowest  average  RMS  error  to  highest  RMS  average  RMS  error)  as  teams  1, 

5,  6, 3  and  10  (upper  half  of  Table  9), 
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Table  9:  Average  RMS  error  for  Exhibition  2  (actual  results  vs.  Bayesian  and  team  projections). 
Each  evaluating  team  (5  total)  evaluated  the  6  participating  teams.  From  these  evaluations, 
Bayesian-updated  projections  and  team  projections  for  Exhibition  2  were  separately  compared  to  the 
actual  results  of  Exhibition  2.  Six  RMS  errors  resulted  in  each  case.  The  6  RMS  errors  for  each  case 
were  summed  and  an  average  taken  with  the  results  displayed  in  this  table. 


Average 

RMS  error  between  actual  results  and  Bayesian  projections  for  exhibition  2 

error  summed 

average  RMS  error 

Team  1 

0.375 

0.063 

Team  3 

1.142 

0.190 

Team  5 

0.423 

0.070 

Team  6 

0.671 

0.112 

Team  10 

1.249 

0.208 

Average 

RMS  error  between  actual  results  and  team  projections  for  exhibition  2 

error  summed 

average  RMS  error 

Team  1 

1.545 

0.257 

Team  3 

1.617 

0.270 

Team  5 

1.523 

0.254 

Team  6 

1.575 

0.263 

Team  10 

0.737 

0.123 

evaluators 

team  1 
-•-team  3 
— Ar-team  5 
—K—  team  6 
— team  10 


teams  evaluated 


Figure  21:  Each  evaluating  team  (5  total)  evaluated  the  6  participating  teams.  From  these 
evaluations,  the  Bayesian-updated  projections  for  Exhibition  2  were  compared  to  the  actual  results  of 
Exhibition  2.  Six  RMS  errors  resulted.  The  6  RMS  errors  associated  with  each  of  the  evaluated 
teams  is  displayed  in  this  figure  for  each  of  the  5  evaluating  teams,  called  evaluators  here.  The  data 
represented  in  this  figure  are  reduced  to  average  RMS  errors  for  each  evaluating  team  in  the  upper 
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half  of  Table  9.  rms  error  =  V 11  i  - 1  ,  with  actual  results  equal 

projections  equal  yi,  and  number  of  bins,  n,  equals  10. 
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Figure  22:  Each  evaluating  team  (5  total)  evaluated  the  6  participating  teams.  From  these 
evaluations,  the  team  projections  for  Exhibition  2  were  compared  to  the  actual  results  of  Exhibition 
2.  Six  RMS  errors  resulted.  The  6  RMS  errors  associated  with  each  of  the  evaluated  teams  is 
displayed  in  this  figure  for  each  of  the  5  evaluating  teams,  called  evaluators  here.  The  data 
represented  in  this  figure  are  reduced  to  average  RMS  errors  for  each  evaluating  team  in  the  lower 
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The  lower  half  of  Table  9  (previous  page)  and  Figure  22  above  provide  a 
tabulation  of  average  RMS  error  and  a  graphical  representation  of  RMS  error  between 
actual  results  and  team  projections  for  Exhibition  2,  respectively.  The  tabulated  data  in 
Table  9  represent  a  sum  of  the  RMS  errors  as  well  as  the  average  RMS  error  for  all  of  the 
teams  evaluated  by  teams  1,  3,  5,  6  and  10  (i.e..  Teams  1, 3,  5, 6  and  10  each  evaluated 
teams  1 , 2, 3,  5, 6  and  10).  An  RMS  error  was  determined  for  each  of  the  6  teams 
evaluated;  this  was  done  for  each  of  the  5  evaluating  teams  (i.e.,  a  total  of  30 
calculations).  An  average  RMS  error,  for  each  evaluating  team,  was  obtained  by  taking 
a  simple  average  of  the  RMS  error  values  (6  total  values)  between  a  team’s  performance 
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in  the  event  (Exhibition  2)  and  the  evaluating  team’s  projections  of  that  team’s  success. 
Figure  22  offers  the  graphical  representation  of  the  data,  which  are  reduced  to  average 
RMS  error  in  the  lower  half  of  Table  9.  Therefore,  the  evaluating  team  (for  Exhibition  2) 
with  the  lowest  average  RMS  error  was  the  team  whose  predictions  for  Exhibition  2  most 
accurately  predicted  the  results  of  Exhibition  2.  For  Exhibition  2,  a  ranking  of  the  teams 
was  determined  (lowest  average  RMS  error  to  highest  average  RMS  error)  as  teamslO,  5, 
1, 6,  and  3  (lower  half  of  Table  9).  This  ranking  is  similar  to  that  described  by  Bayesian 
projection  compared  to  actual  results  except  Team  10  has  moved  from  last  above  to  first 
here  and  teams  1  and  5  above  are  in  reversed  order. 

Of  additional  note  in  Table  9  is  the  fact  that  the  average  RMS  error  is  lower  on 
average  by  0.152  for  teams  1, 3,  5,  and  6  (team  10  exceeds  by  0.085)  for  the  Bayesian 
projection  versus  actual  results  as  compared  to  team  projections  versus  actual  results. 

This  is  an  indication  of  the  fact  that  team  10,  to  this  point,  is  a  fairly  capable  evaluator  as 
it  is  able  to  perform  evaluations  with  less  RMS  error  than  that  obtained  via  Bayesian 
projections. 


6.2.2  RMS  error  for  Competition 

The  upper  half  of  Table  10  and  Figure  23  (both  on  page  70)  provide  a  tabulation 
of  average  RMS  errors  and  a  graphical  representation  of  RMS  error  between  actual 
results  and  Bayesian  projections  for  the  Competition,  respectively.  The  tabulated  data  in 
Table  10  represent  a  sum  of  the  RMS  errors  as  well  as  the  average  RMS  error  for  all  of 
the  teams  evaluated  by  teams  1, 3, 5, 6  and  10  (i.e.,  Teams  1, 3,  5, 6  and  10  each 
evaluated  teams  1,  2,  3,  5,  6  and  10).  An  RMS  error  was  determined  for  each  of  the  6 
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teams  evaluated;  this  was  done  for  each  of  the  5  evaluating  teams  (i.e.,  a  total  of  30 
calculations).  An  average  RMS  error,  for  each  evaluating  team,  was  obtained  by  taking  a 
simple  average  of  the  RMS  error  values  (6  total  values)  between  a  teams  performance  in 
the  event  (the  Competition)  and  the  Bayesian-updated  prediction  of  that  team’s  success. 
Figure  23  offers  the  graphical  representation  of  the  data,  which  are  reduced  to  average 
RMS  errors  in  the  upper  half  of  Table  10.  Therefore,  the  evaluating  team  (for  the 
Competition)  with  the  lowest  average  RMS  error  was  the  team  whose  Bayesian-updated 
predictions  most  accurately  predicted  the  results  of  the  competition.  For  the  Competition, 
a  ranking  of  the  teams  was  determined  (lowest  to  highest  average  RMS  error)  as  3, 5, 1 
and  6  (equal  average  RMS  error),  and  10.  This  ranking  shows  consistency  only  for  teams 
5, 6  and  10.  That  is,  the  data  for  Exhibition  2  (upper  half  of  Table  9)  put  these  teams  in 
the  same  positions  as  seen  here. 
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Table  10:  Average  RMS  error  for  Competition  (actual  results  vs.  Bayesian  and  team  projections). 
Each  evaluating  team  (5  total)  evaluated  the  6  participating  teams.  From  these  evaluations, 
Bayesian-updated  projections  and  team  projections  for  the  Competition  were  separately  compared  to 
the  actual  results  of  the  Competition.  Six  RMS  errors  resulted  in  each  case.  The  6  RMS  errors  for 
each  case  were  summed  and  an  average  taken  with  the  results  displayed  in  this  table. 


Average 

RMS  error  between  actual  results  and  Bayesian  projections  for  competition 

error  summed  average  RMS  error 

Team  1 

1.328 

0.221 

Team  3 

1.069 

0.178 

Team  5 

1.319 

0.220 

Team  6 

1.326 

0.221 

Team  10 

1.355 

0.226 

Average 

RMS  error  between  actual  results  and  team  projections  for  competition 

error  summed  average  RMS  error 

Team  1 

1.173 

0.196 

Team  3 

1.294 

0.216 

Team  5 

1.153 

0.192 

Team  6 

1.064 

0.177 

Team  10 

1.101 

0.183 

evaluators 
—♦—team  1 
—■—team  3 
—k—  team  5 
— >4 —  team  6 
— *—  team  10 


Figure  23:  Each  evaluating  team  (5  total)  evaluated  the  6  participating  teams.  From  these 
evaluations,  the  Bayesian-updated  projections  for  the  Competition  were  compared  to  the  actual 
results  of  the  Competition.  Six  RMS  errors  resulted.  The  6  RMS  errors  associated  with  each  of  the 
evaluated  teams  is  displayed  in  this  figure  for  each  of  the  5  evaluating  teams,  called  evaluators  here. 
The  data  represented  in  this  figure  are  reduced  to  average  RMS  errors  for  each  evaluating  team  in 

the  upper  half  of  Table  10.  rms  error  =  V  i=l  ,  with  actual  results  equal  yi  and  Bayesian 

projections  equal  yi,  and  number  of  bins,  n,  equals  10. 
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teams  evaluated 


evaluators 

—♦—team  1 
—■—team  3 
-a-  team  5 
— K-team  6 
—■—team  10 


Figure  24:  Each  evaluating  team  (5  total)  evaluated  the  6  participating  teams.  From  these 
evaluations,  the  team  projections  for  the  Competition  were  compared  to  the  actual  results  of  the 
Competition.  Six  RMS  errors  resulted.  The  6  RMS  errors  associated  with  each  of  the  evaluated 
teams  is  displayed  in  this  figure  for  each  of  the  5  evaluating  teams,  called  evaluators  here.  The  data 
represented  in  this  figure  are  reduced  to  average  RMS  errors  for  each  evaluating  team  in  the  lower 


half  of  Table  10.  rms  error  =  t n  i  =  1 
equal  yi,  and  number  of  bins,  n,  equals  10. 


,  with  actual  results  equal  land  team  projections 


The  lower  half  of  Table  10  and  Figure  24  above  provide  a  tabulation  of  average 
RMS  error  and  a  graphical  representation  of  RMS  error  between  actual  results  and  team 
projections  for  the  Competition,  respectively.  The  tabulated  data  in  Table  10  represent  a 
sum  of  the  RMS  errors  as  well  as  the  average  RMS  error  for  all  of  the  teams  evaluated  by 
teams  1, 3, 5, 6  and  10  (i.e.,  teams  1, 3,  5, 6  and  10  each  evaluated  Teams  1, 2, 3, 5, 6  and 
10).  An  RMS  error  was  determined  for  each  of  the  6  teams  evaluated;  this  was  done  for 
each  of  the  5  evaluating  teams  (i.e.,  a  total  of  30  calculations).  An  average  RMS  error, 
for  each  evaluating  team,  was  obtained  by  taking  a  simple  average  of  the  RMS  error 
values  (6  total  values)  between  a  team’s  performance  in  the  event  (the  Competition)  and 
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the  evaluating  team’s  prediction  of  that  team’s  success.  Figure  24  offers  the  graphical 
representation  of  the  data,  which  are  reduced  to  average  RMS  errors  in  the  lower  half  of 
Table  10.  Therefore,  the  evaluating  team  (for  the  Competition)  with  the  lowest  average 
RMS  error  was  the  team  whose  predictions  for  the  Competition  most  accurately  predicted 
the  results  of  the  Competition.  For  the  Competition,  a  ranking  of  the  teams  was 
determined  (lowest  to  highest  average  RMS  error)  as  teams  6, 10, 5, 1  and  3  (lower  half 
of  Table  10).  This  ranking  is  consistent  for  all  teams  with  the  exception  of  team  6,  which 
moves  from  second  to  last  position  (lower  half  of  Table  9)  to  the  first  position  here.  Of 
additional  note  in  Table  10  is  the  fact  that  the  average  RMS  error  is  lower  on  average  by 
0.035  for  teams  1, 5, 6  and  10  (team  3  exceeds  by  0.038)  for  the  actual  results  versus 
team  projections  as  compared  to  Bayesian  projection  versus  actual  results.  This  means 
that  the  team  projections  (compared  to  actual  results)  for  all  but  team  3  were  marginally 
better  than  the  Bayesian  projections. 
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Chapter  7 


Overall  observations  regarding  the  contests 

In  Appendix  F  are  found  the  team  rankings  as  evaluated  by  each  of  the  five 
evaluating  teams  (i.e,,  this  is  an  expected  bin  approach  to  ranking  the  teams  as  per 
Equation  of  Chapter  5)  as  well  as  the  results  of  each  of  the  three  events.  In  addition,  the 
Bayesian-updated  projections  of  success  for  Exhibition  2  and  the  Competition  (i.e.. 
Posterior  for  Exhibition  2  and  Posterior  for  the  Competition,  respectively)  are,  likewise, 
shown  in  Appendix  F.  This  Appendix,  like  all  others,  is  for  reference  and  is  simply  Table 
7  with  two  additional  columns  (i.e.,  the  posteriors  for  Exhibition  2  and  the  Competition). 
The  tables  and  figures  in  Chapter  6  are  those,  which  are  used  in  drawing  some 
conclusions  here. 

7.1  Exhibition  1  evaluation  problems 

Because  teams  1, 2, 3, 5, 6  and  10  were  unprepared  for  Exhibition  1,  this  created 
the  necessity  of  using  data,  which  approximated  the  capability  of  these  teams.  These  data 
were  taken  from  the  non-participating  teams  (4,  8, 9,  and  1 1)  as  shown  in  Figure  7. 

These  data  were  fairly  representative  of  the  actual  capability  of  the  participating  teams  as 
evidenced  by  the  RMS  error  between  Bayesian  predictions  of  success  in  events  as 
compared  to  the  results  of  those  events.  Though  the  data  were  representative,  it  would 
have  been  preferable  to  have  data  from  the  participating  teams  as  the  starting  point  for 
these  updating  studies.  If  a  fourth  event  had  occurred  (i.e.,  an  event  following  the 
Competition),  then  the  Exhibition  1  data  could  have  either  been  kept  as  is  and  updated 
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three  times  using  Bayes’  Theorem  or  thrown  out.  The  Exhibition  1  data  would  only  have 
been  thrown  out  provided  Exhibition  2  data  updated  twice  to  predict  the  success  of  the 
teams  in  the  hypothetical  fourth  event  gave  better  results  (i.e.,  lower  RMS  errors). 

Team  6  provided  no  evaluations  for  Exhibition  1.  This  was  problematic  since  no 
Bayesian-updated  predictions  of  Exhibition  2  or  the  Competition  could  be  calculated. 
Instead  of  disregarding  team  6  evaluations  for  Exhibition  2  and  the  Competition,  the  team 
6  evaluations  for  Exhibition  2  were  substituted  for  the,  non-existent,  evaluations  for 
Exhibition  1.  We  believed,  at  first,  that  this  fact  could  have  invalidated  the  Bayesian- 
updated  team  6  evaluations,  which  predicted  the  success  or  failure  of  the  teams  in 
Exhibition  2.  However,  upon  reviewing  the  average  RMS  errors  for  Exhibition  2  as  are 
shown  in  Table  9,  we  note  that  the  average  RMS  error  for  team  6  (for  Exhibition  2)  in  the 
case  of  the  Bayesian  projections  is  third  lowest  (of  five).  In  effect,  using  the  team  6 
evaluations  done  prior  to  Exhibition  2  as  input  (i.e.,  in  place  of  the  non-existent  team  6 
Exhibition  1  evaluations)  for  the  first  use  of  Bayes’  Theorem  (in  Bayesian  updating)  did 
not  adversely  affect  the  team  6  Bayesian  projections  for  Exhibition  2. 

If  the  robots  had  been  more  capable  of  performing  the  required  tasks  in  Exhibition 
1,  then  a  more  rapid  determination  of  the  most  useful  questions  or  questions  that  should 
have  been  included  but  were  not  would  have  occurred.  As  it  was,  questions  that  should 
have  been  tossed  out  like  “Does  the  robot  damage  the  playing  surface?”  remained 
through  the  termination  of  the  Exhibition  2.  Earlier  identification  of  useless  questions 
and  search  for  questions  of  greater  value  in  identifying  distinguishing  characteristics 
between  robots  would  provide  for  a  more  effective  questionnaire  which  would  allow  for 
greater  success  in  predicting  success  of  the  teams  in  future  events. 
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7.2  Accuracy  and  consistency  changes  (from  Exhibition  2  to  the 
Competition)  in  evaluations  performed  by  each  of  the  five  evaluating 
teams 

The  team  rankings,  derived  from  the  Bayesian-updated  evaluation  average  RMS 
errors,  changed  from  the  Exhibition  2  to  the  Competition,  as  is  shown  in  Table  1 1  and  is 
discussed  below.  These  rankings  are,  in  effect,  an  indication  of  how  well  the  teams 
performed  as  evaluators  (i.e.,  the  evaluating  team’s  accuracy  in  evaluation);  the  higher 
the  rank  the  more  accurate  the  team  was  as  an  evaluator.  If  no  new  information  was 
available  between  the  end  of  Exhibition  2  and  the  Competition,  the  rankings  should  have 
remained  consistent.  However,  the  rankings,  in  fact,  take  into  account  the  effect  of  new 
data  between  the  two  events. 


Table  11:  Shows  the  ranking  of  the  evaluating  teams  based  on  average  RMS  error  in  evaluating  all  6 
of  the  evaluated  teams  for  both  Exhibition  2  and  the  Competition.  For  Exhibition  2  and  the 
Competition,  team  rankings  based  on  average  RMS  errors  were  derived  for  the  Bayesian-updated 
team  evaluations  (or  posteriors)  and  the  team  evaluations  (Le.,  not  Bayesian-updated).  The  ranking 
of  the  teams  is  from  the  lowest  average  RMS  error  (best)  shown  at  the  top  of  a  column  to  the  highest 
average  RMS  error  (worst)  shown  at  the  bottom  of  the  column. 
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Note  also  that  it  is  preferable  to  be  the  team  ranked  low  in  Exhibition  2  and  ranked 
significantly  higher  in  the  Competition,  as  it  is  indicative  of  improvement  in  evaluation 
ability,  unless  of  course  a  team  was  ranked  high  throughout  the  competition.  In  which 
case,  that  team  would  have  evaluated  exceptionally  well  from  the  beginning. 

The  team  rankings  from  the  Bayesian-updated  evaluation  average  RMS  errors  are 
as  follows.  Team  1  went  from  the  best  evaluator,  or  most  accurate,  to  third  of  five.  Team 
3  improved  from  fourth  to  first  of  five.  Teams  5, 6  and  10  remained  consistent  at  second 
of  five,  third  of  five,  and  fifth  of  five  (or  least  accurate),  respectively. 

The  team  rankings,  derived  from  the  team  evaluation  (i.e.,  not  Bayesian-updated) 
average  RMS  errors,  changed  as  shown  in  Table  11  and  as  is  discussed  below.  These 
rankings,  just  as  those  discussed  in  the  previous  paragraph,  are  an  indication  of  the 
evaluating  team’s  accuracy  in  evaluation.  The  difference  here  is  that  no  Bayesian 
updating  has  been  performed  on  this  data  set.  Team  1  went  from  the  third  best  evaluator 
to  fourth  of  five.  Team  3  remained  consistent  at  fifth  of  five.  Team  5  slipped  from 
second  to  third  of  five.  Team  6  improved  from  fourth  to  first  of  five.  Team  10  was 
downgraded  from  first  to  second  of  five. 

7.3  Accuracy  and  consistency  changes  (from  Exhibition  2  to  the 
Competition)  in  evaluations  of  the  six  evaluated  teams 


7.3.1  Teams  evaluated  well  as  determined  via  the  Bayesian-updated  team  evaluation 
average  RMS  errors 

Here  an  arbitrary  assumption  was  made  that  an  RMS  error  of  0.1  (or  less)  was 
considered  a  good  evaluation.  For  Exhibition  2,  the  Bayesian-updated  team  evaluation 
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RMS  errors  (for  Exhibition  2)  of  Figure  21  show  that  the  following  teams  were  evaluated 
well  and  by  which  teams  those  evaluations  occurred.  Teams  1  and  5  were  successfully 
able  to  evaluate  team  1.  Teams  1, 5  and  6  successfully  evaluated  team  2,  Teams  1, 3  and 
6  evaluated  well  team  3.  All  teams  (teams  1, 3, 5, 6  and  10)  evaluated  well  team  5. 
Teams  1  and  5  successfully  evaluated  team  6.  Teams  1  and  10  evaluated  successfully 
team  10. 

If  instead  we  now  use  an  RMS  error  of  0.2  in  Figure  21  to  define  a  good 
evaluation  for  the  Bayesian-updated  team  evaluations  of  Exhibition  2,  additional  teams 
are  added  to  the  list  of  good  evaluators.  Teams  3  and  10  successfully  evaluated  team  1. 
Team  5  performed  well  in  evaluating  team  3.  Team  6  evaluated  itself  well.  Teams  3, 5S 
and  6  successfully  evaluated  team  10. 

Here  the  assumption  was  made  that  an  RMS  error  of  0.1  (or  less)  was  considered 
a  good  evaluation.  For  the  Competition,  the  Bayesian-updated  team  evaluation  RMS 
errors  of  Figure  23  show  that  the  following  teams  were  evaluated  well  and  by  which 
teams  those  evaluations  occurred.  Team  3  successfully  evaluated  team  2.  All  teams 
(teams  1, 3,  5, 6  and  10)  evaluated  team  5  well.  The  only  consistency  between  Exhibition 
2  and  the  Competition  was  the  fact  that  all  teams  were  able  to  evaluate  team  5  well.  This 
means  that  each  of  the  evaluating  teams  was  consistently  able  to  identify  a  relatively 
mediocre  (in  performance)  team  (see  Table  8). 

If  instead  we  now  use  an  RMS  error  of  0.2  in  Figure  23  to  define  a  good 
evaluation  for  the  Bayesian-updated  team  evaluations  of  the  Competition,  again 
additional  teams  are  added  to  the  list  of  good  evaluators.  Team  10  performed  well  in  its 
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evaluation  of  team  2.  Teams  1,3,  5,  and  6  successfully  evaluated  team  3.  Teams  1,  5, 
and  6  evaluated  team  6  well.  Team  3  successfully  evaluated  team  10. 

In  the  case  of  0.2  RMS  errors  for  the  Bayesian-updated  team  evaluations, 
consistency  between  Exhibition  2  and  the  Competition  was  as  follows  below.  Teams  1, 
3,  5,  and  6  successfully  evaluated  team  3  for  both  events.  All  teams  evaluated  well  team 
5  in  both  events.  Teams  1, 5,  and  6  successfully  evaluated  team  6  for  the  two  events. 
Team  3  successfully  evaluated  team  10  in  both  events.  Of  note  is  the  fact  that  teams  1, 5, 
and  6  evaluated  effectively  and  consistently  teams  3, 5,  and  6  (all  mediocre  teams  in 
performance).  Team  3,  on  the  other  hand,  was  able  to  evaluate  effectively  and 
consistently  teams  3  and  5  (both  mediocre  teams  in  performance)  and  team  10  (a  high 
performer). 

In  addition  we  should  consider  more  closely,  the  Bayesian-updated  team  3 
evaluation  of  team  2  (an  excellent  performer  in  the  competition).  If  we  discount  the  fact 
that  team  3  had  only  the  fourth  lowest  RMS  error  in  its  Bayesian-updated  evaluation  of 
team  2  prior  to  Exhibition  2  (see  Figure  21),  and  instead  look  at  the  team  3  Bayesian- 
updated  evaluation  of  team  2  prior  to  the  Competition  (see  Figure  23),  we  see  that  team  3 
had  the  best  showing  with  an  RMS  error  of  0.09  and  team  10  was  second  with  an  RMS 
error  of  0.17.  It  is  easier  to  believe  that  a  team  is  a  good  evaluator  if  it  is  capable  of 
evaluating  well  both  good  and  poor  performers.  In  this  case  the  lack  of  consistency 
between  Exhibition  2  and  the  Competition  should  be  ignored,  as  the  Bayesian-updated 
team  3  evaluation  of  team  2  was  extremely  accurate  in  the  Competition,  the  event  that 
counted.  This  makes  team  3  our  choice  for  the  best  evaluator  in  this  instance. 
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7.3.2  Teams  evaluated  well  as  determined  via  the  team  evaluation  average  RMS 
errors 

Again,  the  assumption  was  made  that  an  RMS  error  of  0.1  (or  less)  was 
considered  a  good  evaluation.  For  Exhibition  2,  the  team  evaluation  average  RMS  errors 
of  Figure  22  show  that  the  following  teams  were  evaluated  well  and  by  which  teams 
those  evaluations  occurred.  Team  10  was  successfully  able  to  evaluate  teams  1,  3, 6  and 
10.  Team  3  successfully  evaluated  team  3. 

If  instead  we  now  use  an  RMS  error  of  0.2  in  Figure  22  to  define  a  good 
evaluation  for  the  team  evaluations  of  Exhibition  2,  additional  teams  are  added  to  the  list 
of  good  evaluators.  Team  1  successfully  evaluated  team  3.  Teams  5  and  10  performed 
well  in  evaluating  team  5.  Teams  1, 3, 5,  and  6  successfully  evaluated  team  10. 

Here  the  assumption  was  made  that  an  RMS  error  of  0. 1  (or  less)  was  considered 
a  good  evaluation.  For  the  Competition,  the  team  evaluation  RMS  errors  of  Figure  24 
show  that  the  following  teams  were  evaluated  well  and  by  which  teams  those  evaluations 
occurred.  Teams  5  and  6  successfully  evaluated  team  6.  Team  10  evaluated  itself  well. 
The  only  consistency  between  Exhibition  2  and  the  Competition  was  the  fact  that  Team 
10  was  able  to  successfully  evaluate  itself  over  the  two  events. 

If  instead  we  now  use  an  RMS  error  of  0.2  in  Figure  24  to  define  a  good 
evaluation  for  the  team  evaluations  of  the  Competition,  additional  teams  are  added  to  the 
list  of  good  evaluators.  All  teams  successfully  evaluated  team  1.  Team  6  performed  well 
in  evaluating  team  2.  All  teams  successfully  evaluated  team  3.  Teams  1, 3,  and  10 
successfully  evaluated  team  6.  Teams  1, 5,  and  6  evaluated  team  10  well. 
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In  the  case  of  0.2  RMS  errors  for  the  team  evaluations,  consistency  between 
Exhibition  2  and  the  Competition  was  as  follows  below.  Team  10  successfully  evaluated 
team  1  in  both  events.  Teams  3  and  10  successfully  evaluated  team  3  for  both  events. 
Teams  10  successfully  evaluated  team  6  for  the  two  events.  Teams  1, 5,  6  and  10 
successfully  evaluated  team  10  in  both  events.  This  says  that  team  10  was  consistently 
adept  at  evaluating  four  teams  (1, 3, 6  and  10),  which  were  three  poor  performers  and 
itself  (a  high  performer).  Of  note,  however,  is  the  fact  that  team  10,  a  good  evaluator  as 
well  as  a  high  performer  in  the  events,  did  not  evaluate  its  stiffest  competition,  team  2, 
effectively  (i.e.,  RMS  errors  slightly  greater  than  0.2,  specifically  0.28  for  Exhibition  2 
and  0.23  for  the  Competition).  However,  in  reviewing  Figures  22  and  24,  we  can  see  that 
team  10  was  one  of  the  best  (if  not  the  best)  evaluators  of  team  2,  which  may  imply  that 
the  (arbitrary)  0.2  RMS  error  cutoff  in  determining  if  an  evaluator  was  proficient  may  in 
fact  be  too  low.  Of  additional  note,  is  the  fact  that  three  mediocre  performers  (teams  1, 5 
and  6)  consistently  evaluated  team  10  (a  high  performer)  effectively.  Again,  it  is  easier 
to  believe  that  a  team  is  a  good  evaluator  if  it  is  capable  of  evaluating  well  both  good  and 
poor  performers.  In  this  instance  we  choose  team  10  as  the  best  evaluator. 
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Chapter  8 


The  link  between  the  nuclear  research  and  development 
example  and  the  robot  design  contest 


8.1  The  nuclear  research  and  development  example  compared  to  the 
robot  design  contest 

Twenty-seven  criteria  were  used  in  determining  which  nuclear  power  plant 
concepts  were  well  suited  to  a  given  mission.  The  27  criteria  were  used  in  creating  three 
major  goals  (sustainability,  safety  &  reliability,  and  economics).  That  is,  the  score 
assigned  in  each  of  the  27  areas  was  rolled  into  3  major  scores  called  goals  in  this  study. 
In  the  case  of  the  robot  design  course,  17  criteria  were  used  in  the  creation,  likewise,  of 
three  major  goals  (control,  control/score,  and  score). 

As  is  noted  previously,  problems  existed  in  the  case  of  the  nuclear  example, 
which  do  not  exist  in  the  robot  design  course.  First,  the  work  conducted  was  purely 
hypothetical  (i.e.,  no  power  plants  have  yet  been  built)  and  therefore  no  operational 
comparisons  between  plants  could  be  made,  let  alone  multiple  operational  comparisons  to 
provide  a  fairly  exact  assessment  of  plant  capability.  Second,  inherent  bias  existed 
because  the  concept  designers  evaluated  their  own  designs. 

The  robot  design  course  quite  readily  handled  the  above  two  problems  from  the 
nuclear  example.  First,  the  work  was  carried  to  practical  realization;  the  robots  were 
constructed  and  multiple  (three)  operational  comparisons  occurred  allowing  for  a 


80 


rigorous  assessment  of  each  robot’s  capabilities.  Second,  the  concept  designers  evaluated 
their  robot  as  well  as  those  of  the  competition.  Finally,  these  evaluations  were  then 
compared  to  the  operational  results  of  the  events  (exhibitions  and  competition).  The  fact 
that  the  evaluations  must  stand  next  to  the  results  of  the  events  lessens  the  bias  a  robot 
designer  might  apply  to  his  own  design  especially  in  subsequent  evaluations. 


8.2  Improving  the  nuclear  research  and  development  process  example 
based  on  work  in  robot  design  course 

The  fact  remains,  however,  that  a  more  robust  evaluation  method  for  the  nuclear 
example  (i.e.,  the  impetus  for  this  study)  is  desired.  The  methodology  used  in  the  robot 
design  course  can  be  incorporated  into  the  nuclear  example.  Surveys  (i.e., 
questionnaires)  can  be  conducted  by  independent  evaluators,  from  which,  probability 
mass  functions  concerning  the  performance  variables  can  be  derived.  The  probability 
mass  functions  can  be  used  to  eliminate  those  designs  not  scoring  above  some  threshold 
level.  Further,  more  in-depth  evaluations  can  follow  to  eliminate  other  concepts  until  a 
select  group  of  a  few  concepts  remains.  Obviously,  preliminary  designs  will  become 
more  concrete  as  designs  are  selected  for  further  evaluation. 

How  could  the  surveys  be  conducted?  An  independent  evaluation  team  could 
evaluate  all  of  the  prospective  designs  in  order  to  remove  the  biases  of  individual 
designers  toward  their  own  designs.  The  evaluators’  independence  from  any  of  the 
competing  designs  would  help  to  ensure  that  the  evaluation  is  fair.  The  evaluation  team 
could  conduct  a  survey  (i.e.,  a  series  of  questions)  designed  to  provide  a  probability  mass 
function  predicting  the  success  of  a  given  design.  The  concept  designers  will  have  to 
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justify  (to  the  satisfaction  of  the  evaluators)  answers  to  any  question  and  the  concept 
designers  will  not  be  privy  to  the  scores  assigned.  This  evaluation  process  will  be  done 
in  stages  with  the  threshold  being  raised  for  each  subsequent  evaluation.  This  process 
will  occur  until  a  few  surviving  concepts  remain. 

Following  the  down-selection  to  the  few  surviving  concepts,  construction  of 
small-scale  prototypes  of  the  remaining  concepts  could  occur.  Operational  tests  could 
then  be  conducted.  For  each  operational  test,  after  the  first  test,  Bayesian  updating  could 
be  used  to  predict  the  success  of  each  of  the  concepts  as  done  with  the  robot  designs. 
Bayesian  updating  could  reduce  the  number  (by  one)  of  operational  tests  required  as  it 
can  permit  prediction  of  the  success  of  a  design  in  the  next  event.  Doing  this  could 
reduce  the  number  of  operational  tests  required  for  the  remaining  two  or  three  concepts, 
and  thus,  the  costs  involved  before  final  selection  of  the  winning  concept. 
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Chapter  9 


Conclusions 

9.1  Recommendations  for  future  work  in  improving  evaluation  methods 

9.1.1  Methodology  used  by  teams  in  assigning  a  score  to  each  of  the  teams 

The  methodology  used  by  each  of  the  teams  in  assigning  a  score  to  the  teams  that 
it  evaluated  varied  from  team  to  team.  The  methodology  specific  to  each  team  is  shown 
in  Appendix  G.  The  methods,  in  general  are  described  below. 

1.  The  first  step  is  to  read  the  Wiki  journal  [  3]  daily  to  learn  the  best  practices  of 
other  teams  and  avoid  problems  encountered  by  other  teams.  The  Wiki  Journal,  found  on 
the  MASLAB  website  [  3],  is  a  journal  in  which  the  design  teams  update  their  daily 
progress.  In  addition,  speak  directly  with  the  teams  to  identify  the  best  practices  and  the 
problems  encountered  that  may  have  been  left  out  of  the  journal  entries. 

2.  The  second  step  is  to  observe  other  robot  design  teams  in  the  lab  environment 
and  make  comparisons  to  own  design  and  progress.  Keep  track  of  how  much  effort  is  put 
into  the  robot  as  an  indication  of  expected  success  in  the  upcoming  contests.  Use 
performance  in  previous  events  as  evidence  in  evaluations  of  projected  success  in  future 
events. 

3.  The  third  step  is  to  question  whether  other  teams  often  lack  direction  or 
consistently  know  what  they  are  doing?  Were  other  teams  able  to  build  sensors  on  their 
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own?  The  sensors  are  fairly  sophisticated  and  the  ability  to  readily  employ  them  in  the 
robot  design  was  an  indicator,  potentially,  of  overall  design  sophistication  and  an 
indicator  of  success  in  the  competitive  events.  Were  teams  using  water  jetting  (i.e.,  a 
sophisticated  manufacturing  method)  for  the  robot  chassis?  If  a  team  was  using  such  a 
method  in  the  manufacture  of  the  robot  chassis,  their  robot  was  likely  to  be  fairly 
sophisticated  overall  and  excellent  performance,  thus,  was  expected.  Were  teams 
consistently  making  last  minute  preparations  prior  to  the  contests?  If  so,  poor 
performance  would  likely  be  the  result. 

9.1.2  What  would  have  made  the  evaluation  process  easier  for  the  teams? 

Each  of  the  teams  provided  information  regarding  what  would  have  made  the 
evaluation  process  easier  for  them.  The  team  specific  information  can  be  found  in 
Appendix  H.  A  general  description  of  what  would  have  simplified  the  process  follows 
below. 

Provide  (to  the  evaluators)  questionnaires  better  suited  to  each  event  such  that,  not 
only  a  current,  but  also  a  fairly  exact  assessment  of  each  robot  capability  could  be 
obtained  by  filling  out  such  surveys.  The  capabilities  of  the  robots  were  not  equal  in  each 
of  the  contests.  Additionally  full  capability  was  not  achieved  until  the  final  contest,  the 
Competition,  These  facts  meant  that  some  of  the  questions  asked  in  the  questionnaire  did 
not  apply  until  Exhibition  2  and  possibly  until  the  Competition, 

Spend  more  time  in  filling  out  the  questionnaire.  The  contests  themselves 
offered  the  best  evidence  of  a  robot’s  capability  but  the  first  evaluation  was  based  largely 
on  observations  made  in  the  lab  prior  to  Exhibition  1,  making  for  a  difficult  first 
evaluation.  This  meant  that  more  time  must  be  spent  in  observing  in  the  lab  in  order  to 
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make  the  first  evaluation.  Follow-on  evaluations  must  also  be  based  more  on  the 
observations  in  the  lab  since  the  lab  is  the  second  most  important  opportunity  (next  to  the 
competitive  events  themselves)  to  obtain  the  data  used  in  evaluating  each  of  the  teams. 
Additional  time  could  have  been  spent  in  discussing  problems  as  well  as  best  practices 
and  plans  of  attack  with  each  of  the  competing  teams  and  in  reading  the  Wiki  journal  [  3]. 

Be  consistent  in  filling  out  the  questionnaires.  That  is,  having  previous 
questionnaire(s)  available  could  assist  in  maintaining  greater  consistency  in  answering 
the  questionnaires. 


9.2  What  we  learned  from  the  robot  contests 

9.2.1  Discrepancy  occurred  in  determination  of  which  team  was  the  best  evaluator. 

Combining  the  evidence  and  conclusions  in  Chapter  7,  we  can  now  say  that  two 
teams  were  declared  the  best  evaluators,  but  in  two  different  circumstances  (Bayesian- 
updated  versus  non-Bayesian  updated  team  evaluations).  If  we  adhere  to  the  results 
obtained  from  Bayesian-updated  team  evaluations,  then  team  3  is  the  clear  winner.  Team 
3  consistently  and  effectively  evaluated  teams  3, 5  and  10  where  teams  3  and  5  were  poor 
performers  while  team  10  was  a  high  performer.  However,  if  we  adhere  to  the  results 
obtained  from  team  evaluations  (i.e.,  non-Bayesian-updated),  then  team  10  is  easily  the 
best  evaluator.  Team  10  consistently  and  effectively  evaluated  teams  1, 2,  3,  6  and  10 
where  teams  1,3,  and  6  were  poor  performers  and  teams  2  and  10  high  performers. 
Considering  the  above  facts,  then,  the  best  evaluator  overall  is  team  10  for  the  case  of  the 
non-Bayesian  updated  team  evaluations. 
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Section  9.2.1  details  the  methods  used  by  team  10,  which  gives  some  insight  into 
why  team  10  was  able  to  perform  so  well  as  evaluators.  In  Section  9.2.1  we  note  that 
team  10  had  the  most  complete  and  sophisticated  explanation  of  what  should  be  done  to 
effectively  evaluate  the  robot  teams. 

Why  do  we  see  a  discrepancy  between  the  Bayesian-updated  choice  of  best 
evaluator  and  non-Bayesian-updated  choice?  Potentially  many  reasons  exist.  The 
evaluations  completed  prior  to  Exhibition  1  were  completed  with  minimal  information 
(i.e.,  no  previous  events  had  occurred  by  which  to  judge  the  robots).  Therefore,  the  teams 
were  only  able  to  evaluate  the  other  teams  by  means  of  observations  made  in  the  lab  and 
via  discussions  with  the  other  teams.  Combine  this  fact  with  the  fact  that  the  results  of 
Exhibition  1  were  estimated  because  the  participating  teams  were  not  prepared  for 
Exhibition  1  and  the  Bayesian-updated  team  evaluations  projecting  the  success  of  the 
teams  in  Exhibition  2  may  not  have  made  these  data  the  most  desirable.  In  other  words, 
the  evaluators  went  through  two  steep  learning  curves  in  attempting  to  meet  the 
requirements  of  Exhibition  l(i.e.,  becoming  technically  competent)  as  well  as  attempting 
to  become  proficient  evaluator  of  their  peers. 

Additionally,  the  teams  were  only  minimally  capable  to  compete  by  the  time  of 
Exhibition  1  and  for  that  matter,  that  of  Exhibition  2  as  well.  It  would  have  been 
preferable  to  have  had  a  fourth  event  to  allow  slightly  less  advanced  teams  to  make  the 
changes  that  would,  perhaps,  have  changed  their  robot  from  a  poor  performer  to  a  good 
or  even  high  performer. 

Examples  of  the  previous  statement  include  the  non-official  results  of  robots 
competing  immediately  following  the  Competition.  The  robot  teams  (not  all  teams 


86 


participated)  made  minor  improvements  after  that  team’s  performance  in  the  Competition 
and  were  allowed  at  the  conclusion  of  the  competition  to  compete,  unofficially.  Team  3 
scored  4  points  (They  had  scored  0  points  in  the  Competition.).  Team  4  (one  of  the  teams 
not  participating  in  our  project)  scored  8  points  (They  had  scored  0  points  in  the 
Competition.).  Team  9  (another  team  not  participating  in  our  project)  scored  13  points, 
which  is  one  more  point  than  the  winning  team  (team  10)  scored  in  the  Competition. 

In  other  words,  robot  progression  from  one  event  to  another  was  more  of  an 
exponential  change  rather  than  a  linear  change.  This  fact  makes  evaluation  harder  for  the 
novice  evaluator,  although  team  10  was  able  to  perform  extremely  well. 

9.2.2  Methodology  of  the  best  evaluator  (team  10) 

As  part  of  this  work  it  was  important  to  understand  better  how  qualified  experts 
evaluate  technological  concepts.  The  methodology  used  by  team  10  in  evaluating  the 
robot  teams  prior  to  the  three  events  follows  below.  Note  that  team  10  had  the  most 
complete  and  sophisticated  (among  the  participating  teams)  idea  or  explanation  of  what 
should  be  done  to  effectively  evaluate  the  robot  teams. 

The  method  that  team  10  used  included  the  following  steps. 

1.  Observe  the  robot  teams  in  the  lab  environment.  Observe  whether  other  teams 
often  lacked  direction  or  consistently  know  what  they  are  doing?  Were  other  teams  able 
to  build  sensors  on  their  own?  The  sensors  are  fairly  sophisticated  and  the  ability  to 
readily  employ  them  in  the  robot  design  was  an  indicator,  potentially,  of  overall  design 
sophistication  and  an  indicator  of  success  in  the  competitive  events.  Were  teams  using 
water  jetting  (i.e.,  a  sophisticated  manufacturing  method)  for  the  robot  chassis?  If  a  team 
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was  using  such  a  method  in  the  manufacture  of  the  robot  chassis,  their  robot  was  likely  to 
be  fairly  sophisticated  overall  and  excellent  performance,  thus,  was  expected.  Were 
teams  consistently  making  last  minute  preparations  prior  to  the  contests?  If  so,  poor 
performance  would  likely  be  the  result. 

2.  Observe  of  team  performance  in  the  competitive  events.  This  is  the  most 
obvious  and,  likely,  the  most  useful  means  by  which  the  teams  should  be  evaluated. 
Performance  in  previous  events  is  good  evidence  of  the  projected  success  of  the  teams  in 
future  events. 

3.  Identify  problems  and  best  practices  of  the  other  teams.  This  can  be 
accomplished  by  reading  the  Wiki  Journal  [  3],  an  online  source  of  information,  and 
discussing  these  concerns  with  the  other  teams.  When  team  10  was  interviewed  for  their 
response  as  to  what  their  method  of  evaluation  consisted  of,  they  did  not  state  that  they 
had  used  the  Wiki  Journal.  Although  team  10  did  not,  specifically,  indicate  that  they  had 
referred  to  the  Wiki  Journal,  we  assume  that  they,  in  fact,  did  so  as  did  all  the  other 
teams.  In  addition  to  using  the  Wiki  Journal,  we  likewise  assume  that  team  10  also  spoke 
with  the  other  teams  to  identify  problems  and  best  practices  that  may  not  have  been 
included  in  the  Wiki  Journal. 

9.2.3  Success  of  the  probabilistic  approach  to  predicting  the  performance  of 
technological  concepts  (i.e.,  the  robots) 

In  addition  to  understanding  better  how  qualified  experts  evaluated  technological 
concepts,  we  wanted  to  determine  if  a  probabilistically  formulated  method  of  integrating 
knowledge  of  various  performance  attributes  provides  better  understanding  of  the  likely 
performance  of  a  technological  concept.  This  was,  in  fact  accomplished.  Chapter  3 
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describes  how  the  data  (projections  of  the  team’s  success)  were  obtained  via 
questionnaires.  Chapter  4  details  how  weightings  were  assigned  to  the  data  (i.e.,  how 
probabilities  were  assigned  identifying  the  likelihood  of  a  team’s  success  in  specific 
performance  areas).  Chapter  5  compares  the  data  to  the  results  of  the  three  events. 
Chapter  6  shows  the  errors  between  the  team  projections  of  success  and  the  actual  results. 
Chapter  7  shows  that,  although  some  teams  were  better  evaluators  than  others,  successful 
evaluations  of  the  teams  occurred.  Therefore,  a  probabilistically  formulated  method  of 
integrating  knowledge  of  various  performance  attributes  can  provide  a  better 
understanding  of  the  likely  performance  of  a  technological  concept,  in  our  case  robot 
performance. 

9.3  Lessons  which  can  be  extended  to  the  broader  concern  of  how 
companies  allocate  R  &  D  funding,  to  include  the  acquisition  of  a  new 
Naval  vessel  or  targeting  technology  investments  in  the  drug  delivery 
process. 

Whatever  the  various  options  are  for  which  R  &  D  funding  can  be  allocated  (e.g., 
the  newest  Naval  vessel  or  technology  investment  in  the  drug  delivery  process),  a 
thorough  and  sophisticated  evaluation  of  those  options  should  occur.  The  overwhelming 
success  of  team  10  in  evaluating  effectively  five  of  six  participating  robot  teams  was 
largely  due  to  the  thorough  and  sophisticated  process  used  in  evaluating  the  robot  teams. 
Yes,  this  could  go  without  saying  and,  yet,  the  other  four  evaluators  (teams  1, 3, 5  and  6) 
had  lesser  degrees  (some  significantly)  of  success  in  their  evaluations. 

Both  the  Naval  ship  acquisition  process  and  the  technology  investment  in  the  drug 
delivery  process  specified  phases  that  must  be  passed  through  to  effectively  obtain  the 
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best  ship  and  the  technology  which  will  best  be  used  to  increase  research  productivity, 
respectively.  Within  these  phases  used  to  ultimately  select  and  produce  the  desired 
outcomes,  lies  the  opportunity  to  employ  a  probabilistically-formulated  method  of 
integrating  knowledge  of  various  performance  attributes  that  can  provide  a  better 
understanding  of  the  likely  performance  of  the  technological  concept,  in  our  two 
examples  the  ability  of  the  ship  to  meet  the  mission  need  requirements  and  the  ability  to 
increase  research  productivity  (ship  acquisition  and  targeting  technology  investment  in 
the  drug  delivery  process,  respectively). 

First,  surveys  should  be  created  by  the  individuals  providing  the  requirements  that 
must  be  met  by  the  new  ship  design  or  new  technology  used  to  improve  research 
productivity.  Surveys  can  be  completed  at  significant  milestones  or  phases  in  the 
process.  The  surveys  at  the  start  of  a  specific  milestone  can  be  compared  to  the  results 
obtained  at  the  end  of  that  phase.  Bayesian  updating  (or  the  use  of  Bayes  theorem)  can 
be  used  to  predict  the  future  success  of  the  ship  design  or  the  new  technology  at  the  end 
of  the  next  phase.  If  only  one  design  or  new  technology  is  under  consideration,  the 
Bayesian-updated  prediction  may  very  well  produce  the  data  used  to  determine  if  the 
process  will  continue  or  if  the  results  do  not  justify  continued  support.  If  multiple 
designs  or  technologies  are  considered,  the  Bayesian-updated  prediction  may  well  be 
used  to  down-select  one,  or  more,  of  the  competing  designs  or  technologies  at  any  phase 
in  the  selection  process. 

Bayesian  updating,  in  effect,  can  be  used  to  limit  the  resources  allocated  to 
specific  designs  or  new  technologies  by  shortening  the  selection  process.  The  Bayesian 
updating  forecasts  the  success  of  the  design  or  new  technology  in  the  next  phase.  If  that 


forecast  does  not  meet  the  threshold  requirement  for  that  phase,  then  the  funding  for  that 
design  or  new  technology,  not  meeting  the  requirements,  would  be  terminated.  Only 
those  designs  or  new  technologies  meeting  the  threshold  requirement  for  that  stage  of  the 
process  would  be  allowed  additional  funding. 
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Appendix  A:  Exhibitions  1  &  2  questionnaire 

The  questionnaire  used  in  Exhibitions  1  and  2  is  presented  below.  The  questionnaire  shows  the  questions  used  to  project  a 


Fill  out  one  for  your  team  and  then  one  for  each  of  the  participating  teams.  The  participating  teams  are  1, 2, 3, 5, 6, 10 
Team  members  should  split  the  work  so  only  2  or  3  questionnaires  maximum  will  be  filled  by  any  one  person. 
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Appendix  B:  Data  collected  from  Exhibition  1 

Evaluations  were  performed  by  teams  1, 3, 5,  and  10  for  Exhibition  1.  All  of  the  following  applies  to  these  teams  but  not  to 
team  6,  which  did  not  provide  evaluations  for  Exhibition  1.  Team  roll-up  evaluations  prior  to  Exhibition  1  is  the  major  heading  for 
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RMS  error).  This  is  where  the  comparison  between  actual  results  and  the  Bayesian-updated  evaluations  is  made. 


The  next  major  heading  is  difference  squared  between  actual  results  and  team  projections  for  Exhibition  2  (in  terms  of  RMS 
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Appendix  D:  Competition  questionnaire 


to 

in 

*  c 
*?  O 


c 

o 


2 

J3 


O 

C 


8 

(8 


2 

a 

re 

8 

a 

re 

re 

£ 

05 

C 

o 

■o 

2 

re 

o 

£ 

5 

1 

T“ 

to 

o 

2 

c 

05 

"□ 

3 

% 

>* 

£ 

** 

3 

O 

*D 

O 


re 

XI 

s 

2 

re 

c 

c 

o 

3 

re 

0) 


re  a 

"  i 
i  § 

1  I* 

g  £ 

I  ^ 

jo.  sr 
o  *- 
t=  re 

i  ^ 

a  = 
*  * 


it 

1  i 

re 

£  2 

2  re 
5*  c 
re  g 
■e  .2 

H  » 
a.  a) 

o  3 
£  O' 

z  « 
I  S 

o  « 

s  t 

■2  S 

re  re 

g  ^ 

o  t- 

g  i 
§  i 

*o  «- 
c  ^ 

*  «■ 
E  re 

§  2 

•M  3 

I  ■= 

k.  2 

a  | 

i  I 

i  i 

s  I 

o  E 
—  re 
“  re 
u-  H 


O 

c 

o 


re 

o. 

E 

8 

re 


S 


c 

re 


re 

2: 


E 

re 

E 

i 

re 

re 

o 


1 


3 

O 


3= 

W  © 


I 


•C 

o 

£ 

5 

w 

O 


re 

X3 


3 

C 


re 

re 


re 

p 


re  f 
o 

c  _ 
re  fs 

§  i  1 

re  g  H. 

w  £  cr 

re  t:  S 
>  re  2 
re  o.  w 


c 

re 


8 

re 

§ 

8 

E 

*■3 

V 

8 

re 

c 

re 

re  m 


re 

c 

c 

o 

■►■> 

re 

re 

3 

O' 

re 


3 

O 

05 


£ 

I 


o  2 

£  § 
re  re 
re  re 

1  « 

2  § 

ji 

o  o 
if,  to 


re 

05 

C 

£3 


re 

re 

re 

re 


5 

f— « 


select  peak 


robot  begins  computing  or  starts  average  begins  moving  at  exactly 

prior  to  10  second  required  delay  performance  10  seconds 

(moves  starting  between  30  and  10  sec) 


CM 


C 

E 

co 

>. 

o  5  o 
3  =  ^ 

S  0 

<0  c 
Q.  Jr 

O  +- 

5  i 

o  |  O) 
-D  O 


CO 

CO 

i  g  0 

.c  '5  ° 

S  I 
®r  S 

=-  S 


|  8 
CO  CO 


o 


s 

§  E 
m  o 

C  a 
O) 

OT  5 

c  co 

!  ? 

§  c 

CO  £ 

1 1 

£  ©  <*> 

if 


CO 

CD  Q. 
o  CO 

m  2 
£  ©  10 
o  I 

t:  - 
a)  £ 

CL  £ 

o  5 

*  £ 

2  8* 

5  o 

%  e 


0) 

o 

c 

CO 

fl>  c 

D)  E 
2  O 
©  t 

>  2 
CO  Q. 


© 

o 

c 

CO 

0)  c 
co  E 
2  o 
<u  t 

>  £ 
co  a. 


co 

c 

o> 


CM 


G) 

O)  -* 

£  S 

2  Q. 

•M 

o  o 
©  a) 
a)  © 

CO  CO 


c 

£ 

5  I 

CO  = 
CO  0) 
2  .1 

1 

c  © 
s  & 

£  o 

2  CO 


© 

g  « 

2  2. 


a 

© 

© 

co 


£ 

©" 


2  I 


O 

c 


«  1- 


<0  Jr 

»  =  I 

§  «  | 

n  ro  > 


© 

g  « 

5  8. 


O 

© 

© 

© 


f  e  I 

5  s  g 

i  4  * 

£  —  ® 

S  w  s 
8-  |  8  ^ 

1  I  & 
c  ©  >. 

W  i.  c 
©  ©  w 
O  31  +* 

T5  «  CO 


© 

O) 

c 

2 

4-» 

a 

© 

© 

© 


select  peak 


retrieves  and  places  in  yellow  average 

scoring  locations  far  less  than  performance  retrieves  and  places  in  yellow  areas 

average  #  of  targets  greater  than  average  #  of  targets 


CD 


O 


g  ® 

I  B  O 

S  I  - 

£  8 

ra  to 

1  3 
<5  o 

o  *©  o* 


oi 


o 

8  8 

€  £ 
15  £Z 
a)  « 

■D  O 

%  §■ 

i  - 

■=  p 


o  " 
2  B 


m  » 
£  © 
o  2> 
o  « 

*  z 

©  o 
E  % 
O  © 
o 

C  IE 
© 


3 


Me© 

m 


a 

•a 


1  i 


>  Ol 

.2  „  o> 

S  £ 


a) 

ai  * 

§  £ 

o  o 
©  © 
©  © 
to  to 


c 

IS 

©  p 

i 

©  € 
§  a 


©  M 
2 

"S  © 
o>  m 
o  .5 

+•*  w 

©  o 
o 

jQ  © 

g  3  ” 

3  O 

I  1 

O  © 


© 

3»  •* 

§  8. 

I  | 
©  © 
to  © 


© 

o 

c 

© 

©  c 
©>  = 
P  o 

©  r 
>  © 
©  CL 


©  _  (O 

e>  g 
s  i 
|  8  w 

1  I 

O  © 

++  >> 

©  c 
xs 

?r 

2  a 

5  * 


SO 


P  O 

w  ■*-» 


a  * 

£  fS 

§  8. 

4-*  ■*-* 

1  8 

©  © 

©  © 


m 


©  c 

g»  E 

2  o 
©  ^ 

3  8. 


O 

©  *•** 

©  SL 

SCO 
© 

I  > 

«  I 

8  f 

Q-  © 

13  JS 

ife 
I  §  S 
s  |  | 

I  S  I 

£  o  o 


© 

Of} 


select  peak 


when  robot  fails  to  grasp  target  average  when  robot  falls  to  grasp  target 

not  smart  enough  to  try  again  performance  Is  smart  enough  to  try  again 


CO 

Q. 

§ 

c  O 

O  — 

c  -P 


1  « 

>  a 

S  js 

•o  o  ^ 

=  Q)  O) 

o 

o  o 


CO 

(0 

Si 

(0 

o>  © 


X5 

!  8 

^  CO 

2  o> 

?  .E 

2  3 
t  I 


■Q  S'  o 
5  «  ■»- 


o 

o 


o> 


■go  _ 
O  0)  O) 

1 1 


r- 


c 

CO 

d)  c 

o>  E 

2  o 
cd  t 

>  2 
co  Q. 


d> 

o 

c 

CO 

CD  C 

O)  E 
2  o 

CD  t 

>  2 
CO  Q. 


o> 

c 


CD 

O 

c 

CO 

©  c 

2  ^ 
2  o 
o>  t: 

>  £ 
CO  Q. 


CM 


0) 

O) 


CO 


I 

a> 

■D 


CD 

S>  * 

=  s 

2  a 

o  o 
o)  a> 
o  a> 
v>  V) 


T3 

j-  ® 

1  J5 

>  Q. 

©  £ 

■a  o  ^ 

=  .®  £ 

O  -Q  co 

O  O  CL 


a> 

■g  » 
JS  co 

CD  2 

c  ■ 

3  g> 


CD 

■o 

C 


43 

CO 


CD 

c  « 
§  8. 
■*-»  +- 
O  O 
©  © 

©  © 
CO  cn 


43  O 
O  o 

t.  CA 


O 

D) 

C 


o 

© 

Q) 

CO 


£  |  - 
C  ® 

3  ff 
O  S 

-D  <D 

O  £ 


CD 

§  8. 

o  o 
a)  a> 
a>  cd 

V)  CA 


robot  has  long  processing  time  in 

detecting  scoring  areas  average  robot  has  short  processing  time  In 

and/or  targets  performance  detecting  scoring  areas  and/or  targets 


c 


o 
re 

Ui 

a 

c 

©  _ 

O  £ 

©  CL 

*  s 

%  2  05 

•  £ 


re 

£ 

o 

£ 

§  I 
£  a 

t  w 
w  o 


re 


re 

a 

m  ° 

S 

1 

C 


o 

£ 


©  0> 


2  i 


8.  o 

o  > 

if® 

£  — 

S  2 


oo 

Tt* 


re 

o>  ■£ 


re  re 
re  re 
re  re 


re 

re 

c 

re 

re  c 

W  E 

2  o 
re  £ 
>  re 
re  o. 


U) 

c 

re 

a 

M— 

o 

re 

D) 

1  1 
re  2 

2  * 
re  a 

=  8 
re  15 
£  re 
re  £ 


re 

g  « 

I  a 


re 

o 

c 

re 

re  c 
op  E 
re  o 

re  t: 
>  re 
re  a. 


re 

E 

o 

£ 

re 

!  i 


i  i 

I! 


c 

re 

re  re 

o>  E 

re  o 
re  H 
>  re 
re  Q. 


>. 

£ 

re 


1 

2 

a  re 
o  5 

1  | 
e  C 
re  re 

re  £ 
« 
£ 


E 


re 

m  ■* 

g  s 

5  o. 


P  § 


o 

re 

re 

re 


o 

re 

re 

re 


re 

i>  « 
2  8. 
<+* 

©  © 

re  re 
re  re 


robot  team  1  is  last  In  competition  average  robot  team  1  wins  competition 

performance 


Os 


CM 


<D 

1  S 

2  8. 

4-* 

jj  8 

a>  o 

CO  CO 


s 


O  C 

g>  E 

2  o 

q>  t; 

>  £ 
co  a. 


a> 

Q. 

£ 

o 

o 


CO 

re 


CM 

E 

co 

0) 

o 

o 


<d 

g  is 

2  8. 

■H  4~* 

8  8 
V  CD 
CO  CO 


© 

O 

c 

CO 

<D  C 

g>  £ 
2  o 
©  t; 
>  © 
to  a. 


c 

o 


8. 

E 

o 

o 


co 

TO 


CO 

E 

CO 

G) 

HM 

o 

-Q 

o 


o 

CM 


1  « 

2  8. 

4-*  4-1 

8  8 
a>  © 
co  co 


© 

o 

c 

(0 

©  c 

g>  = 
2  o 
©  t: 

>  £ 
CO  Q. 


© 

Q. 

E 

o 

o 

c 


CM 


E 

CO 

© 

4-* 

4-» 

o 

13 

O 


g  » 

2  8. 

4-»  4—* 

8  8 

O  0) 

CO  CO 


CM 


robot  team  6  is  last  in  competition  average  robot  team  6  wins  competition 

performance 


o 


ft) 

o 

c 

<3 

ft)  E 

E 

2  o 

o 

>  $ 
rg  o. 


ft) 

o. 


CM 

CM 


8)  « 

I  I 

I  I 

0)  ft) 
»  « 


o 

o 


ft) 

o 

T“ 

E 

ro 

a 

4-* 

o 

n 

o 


CM 


CM 


8.  - 

s  I 

ik-  o. 

■4-»  4-* 

M  M 

ft)  ft) 

0)  to 


for  this  event. 


C  (control)  Bins  1-10 

123456789  lOchecksum 

Teaml  0  0  0  0  0  0  0.16  0.48999  0.18799  0.16199  0.99997 


h- 

h- 

N 

CM 

T- 

N- 

1— 

1— 

05 

T— 

c 

CO 

in 

05 

T““ 

T~ 

T— 

05 

05 

05 

T — 

05 

L. 
— * 

05 

05 

L. 

05 

05 

05 

CD 

05 

05 

CD 

05 

C/5 

o 

05 

05 

3 

CO 

o 

05 

05 

05 

o> 

05 

05 

o 

05 

05 

05 

o 

o 

05 

o 

d 

05 

O 

05 

05 

d 

o 

O 

0 

0 

o 

£ 

-C 

o 

o 

CD 

CO 

05 

CO 

o 

co 

o 

CM 

o 

r- 

CM 

o 

O 

o 

in 

T— 

in 

in 

05 

CO 

05 

05 

00 

o 

CO 

rt 

o 

o 

05 

05 

in 

CO 

T — 

00 

00 

Q 

o 

o 

O 

05 

X 

o 

o 

o 

o 

o 

CM 

O 

T— 

d 

d 

d 

o 

o 

o 

o 

w 

o 

o 

o 

d 

05 

05 

05 

in 

CO 

05 

00 

o 

V- 

o 

05 

N- 

05 

O 

o 

m 

05 

T— 

05 

o 

05 

CO 

CO 

LO 

CO 

CM 

o 

■ 

LO 

o 

X 

o 

o 

o 

05 

X 

o 

o 

1 — 
o 

T— 

o 

8 

CM 

o 

CM 

o 

o 

o 

o 

o 

d 

o 

o 

05 

in 

05 

Is* 

00 

CO 

o 

h- 

o 

in 

r^ 

00 

O 

CM 

in 

m 

05 

00 

05 

in 

00 

o 

m 

o 

LO 

O 

CO 

CM 

• 

• 

CM 

q 

X 

CO 

o 

05 

X 

CO 

o 

CO 

o 

o 

o 

d 

d 

o 

o 

d 

CD 

o 

o 

d 

o 

o 

CO 

CO 

s 

05 

05 

h*  O  O  h-  O  1- 

N  O  CM 

in 

00  1-  1- 

o 

W 

h- 

Is- 

o  o 

o 

05 

o 

CO 

in 

CO 

CO 

O 

05 

o  o 

o 

CM 

o 

T_ 

o  o 

o 

o 

o 

o 

o 

o 

o 

O  1^ 

00 

'M' 

00 

CO 

h- 

CM 

CM 

h- 

CO  CO 

m 

00 

in  t- 

00 

00 

r — 

o 

i— 

m 

o 

in 

in 

o 

CO 

CM 

O  fX 

00 

d 

o 

o 

o 

o 

o 

d 

d 

T — 

o 

o 

o 

o 

d 

d 

o 

o 

d 

w 

o 

00 

in 

o 

CM 

CM 

Q 

in 

CM 

CM 

00 

T— 

T— 

o 

m 

00 

h- 

in 

in 

00 

o 

CO 

CO 

CO 

N- 

o 

N- 

o 

o 

T" 

1 

d 

o 

CM 

o 

T— 

d 

CM 

o 

CM 

o 

w 

o 

T - 

d 

N- 

o 

d 

o 

o 

i — 

d 

d 

d 

in 

in 

d 

c 

c 

5 

5 

r-. 

o 

05 

0.44 

1^- 

CO 

05 

m 

T- 

i — 

CO 

in 

C5 

o 

00 

o 

CO 

CO 

o 

s 

o 

d 

T~ 

o 

CM 

o 

CM 

o 

CM 

o 

o 

d 

o 

d 

T— 

1 — 

o 

o 

d 

o 

d 

o 

d 

d 

00 

CO 

o 

00 

00 

"o 

CO 

o 

05 

CO 

CO 

CO 

CO 

CO 

— 

in 

in 

CM 

T— 

1 — 

00 

d 

o 

o 

o 

d 

T- 

o 

d 

o 

o 

V) 

05 

CO 

o 

o 

o 

o 

o 

d 

o 

o 

oT 

i_ 

o 

C0 

05 

O 

CO 

CD 

CO 

h- 

T“ 

1 — 

o 

o 

d 

d 

d 

o 

d 

o 

d 

o 

in 

d 

^B 

o 

^r 

o 

c 

o 

CM 

o 

05 

CO 

CD 

CO 

CO 

o 

Lb 

CM 

i — 

m 

05 

T_ 

7_ 

o 

d 

o 

o 

■t — 

o 

d 

o 

! 

w 

05 

05 

O 

o 

o 

d 

o 

d 

o 

d 

o 

d 

*-» 

C 

o 

o 

CO 

CO 

CM 

d 

CO 

CO 

o 

o 

o 

o 

o 

d 

o 

c 

6) 

d 

5 

c 

o 

o 

tj* 

a 

T” 

o 

05 

CO 

o 

CO 

CO 

-v — 

in 

05 

i — 

T — 

T- 

i — 

1 - 

T — 

> 

05 

o 

o 

o 

Jr 

CO 

CO 

O 

o 

_r  * 

• 

o 

d 

o 

d 

o 

o 

CO 

in 

T— 

o 

o 

o 

o 

o 

in 

h- 

co 

CO 

CM 

o 

o 

o 

o 

d 

d 

O 

CM 

d 

CO 

(0 

O 

o 

cn 


CM 

CO 

in 

CO 

o 

T- 

CM 

CO 

in 

CO 

o 

CM 

CO 

in 

CO 

O 

E 

E 

E 

E 

E 

E 

£ 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

ai 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

O 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

H 

H 

H 

H 

h* 

1- 

H 

H 

H- 

h- 

h* 

h- 

h- 

1- 

1- 

S  (score)  Bins  1  -10 

1234567  89  lOchecksum 

Teaml  0.43706  0.27306  0.10906  0  0  0.03  0.09  0.03399  0.00999  0.01599  0.99915 

Team  2  0.144  0.432  0.144  0.03  0.104  0.092  0.08  0.038  0.006  0  1.07 

Team  3  0.074  0.08  0.0855  0.0915  0.0981  0.1101  0.0951  0.1116  0.1218  0.1308  0.9985 


ID 

C 

CD 

T“ 

T* 

ID 

ID 

00 

05 

05 

05 

05 

05 

O 

05 

05 

05 

05 

o 

05 

O 

o 

05 

o 

E 

ID 

ID 

5s- 

CO 

05 

T“ 

Is- 

05 

y— 

0 

m 

05 

CO 

05 

05 

JC 

05 

05 

05 

05 

o 

CO 

CD 

o 

o 

ID 

CM 

ID 

ID 

ID 

O 

a> 

x 

05 

a> 

o 

o 

CO 

O 

T“ 

CD 

T— 

o 

CO 

CM 

O 

0 

w 

X 

05 

o 

CO 

CM 

CO 

d 

d 

o 

o 

O 

q 

o 

XI 

O 

o 

CO 

w 

o 

ID 

Is- 

CD 

T— 

d 

d 

O 

o 

05 

O 

CD 

05 

o 

ID 

O 

CO 

h- 

d 

d 

o 

q 

CD 

CD 

CD 

05 

ID 

CD 

ID 

ID 

ID 

d 

d 

o 

CO 

CO 

CO 

o 

CO 

CM 

O 

o 

o 

d 

CO 

T— 

o 

CM 

f— 

d 

o 

d 

o 

O 

o 

T*~ 

o 

o 

d 

05 

ID 

o 

ID 

CO 

CO 

CD 

£ 

o 

O 

o 

CO 

05 

d 

CO 

O 

d 

q 

q 

CO 

y— 

CD 

CO 

ID 

CD 

T~ 

ID 

ID 

ID 

d 

d 

'Sf 

CM 

T~ 

w 

CD 

T — 

CO 

CM 

O 

O 

T~ 

d 

CM 

T“ 

r* 

f— 

t- 

CM 

CM 

o 

o 

o 

CO 

q 

d 

CD 

ID 

ID 

ID 

o 

o 

(/) 

o 

d 

CD 

Is- 

CO 

C 

o 

o 

T— 

o 

CD 

s 

d 

d 

00 

o 

o 

d 

o 

CD 

f— 

r* 

h- 

T“ 

CD 

CD 

ID 

ID 

ID 

d 

CM 

ID 

T — 

O 

CD 

Is- 

o 

o 

o 

o 

05 

q 

o 

d 

o 

CM 

d 

q 

d 

h- 

o 

ID 

CD 

CO 

T“ 

d 

d 

d 

r* 

h- 

CM 

o 

CM 

T— 

T““ 

o 

O 

W 

o 

d 

3 

CD 

CO 

CD 

*— 

ID 

CD 

ID 

T“ 

* 

d 

d 

CD 

o 

X 

O 

CO 

h- 

X 

o 

CM 

d 

CD 

O 

o 

s 

o 

o 

m 

m 

o 

3 

o 

w 

CM 

o 

d 

CD 

T— 

d 

w 

CD 

§ 

ID 

CO 

ID 

ID 

s 

T— 

05 

h- 

o 

d 

05 

CM 

T— 

T— 

CM 

CD 

CD 

05 

ID 

o 

CM 

ID 

T- 

ID 

ID 

d 

T*” 

4” 

o 

T— 

d 

’T— 

o 

CO 

CM 

■  C 

ID 

X 

h- 

O 

o 

T— 

o 

CO 

o 

o 

CD 

O 

o 

0 

0 

5 

0 

d 

CO 

d 

w 

CD 

CO 

d 

o 

O 

10 

CM 

ID 

ID 

ID 

CM 

T— 

CO 

CO 

CO 

ID 

CD 

CM 

CM 

00 

T“ 

y— 

CO 

05 

•j 

0" 

o 

ID 

▼“ 

o 

ID 

ID 

i— 

0 

d 

o 

CO 

d 

d 

T“ 

d 

CO 

05 

o 

CD 

(X 

Is- 

o 

£ 

o 

o 

O 

d 

0 

O 

w 

O 

CM 

d 

s 

d 

d 

d 

ID 

ID 

CD 

T~ 

05 

O 

CD 

o 

O 

ID 

ID 

C 

’55 

O 

d 

T — 

d 

05 

ID 

d 

t- 

d 

T — 

d 

o 

h- 

CO 

o 

ID 

o 

o 

ID 

ID 

* 

o 

CD 

T- 

Is- 

CD 

O 

o 

£ 

o 

CO 

r— 

d 

d 

0 

d 

CO 

h- 

ID 

ID 

Tf 

05 

05 

o 

o 

T“" 

CM 

CM 

o 

Is- 

Is- 

ID 

CD 

T“* 

t— 

o 

o 

*■«-** 

CD 

O 

Is- 

CO 

05 

O 

d 

d 

d 

o 

CO 

05 

O 

CM 

o 

o 

o 

o 

o 

ID 

CD 

d 

o 

d 

O 

o 

o 

d 

CM 

CM 

h- 

ID 

CM 

ID 

05 

05 

O 

CM 

h- 

CM 

h- 

Is- 

CO 

CD 

O 

O 

o 

O 

o 

CO 

0 

o 

o 

o 

o 

o 

ID 

CD 

T — 

5 

d 

d 

d 

d 

CD 

co 

w. 

CD 

o 

q 

o 

o 

O 

CD 

o 

d 

d 

o 

0 

CM 

05 

d 

T" 

h- 

ID 

CM 

h- 

05 

05 

*0 

C 

CM 

h- 

o 

Is- 

h- 

c 

*nz 

-r- 

CD 

o 

o 

o 

O 

0 

C5 

CD 

CM 

CD 

CD 

d 

d 

d 

d 

0 

d 

O 

is 

d 

0 

4-i 

o 

o 

c 

o 

ID 

CD 

CM 

CO 

ID 

CD 

o 

4— 

CM 

CO 

ID 

CO 

T“ 

£ 

E 

E 

E 

E 

E 

E 

E 

E 

Q 

E 

E 

E 

E 

E 

E 

0 

0 

0 

cd 

cd 

<d 

0 

0 

0 

CD 

0 

0 

0 

0 

0 

0 

0 

CD 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

H 

H 

{— 

H 

H 

H 

H 

H 

H 

U 

1- 

H 

H 

H 

1- 

i- 

Goals  /  weighting  Complete  goal  rollup  of  Bins  1-10 

Control 


c 

CO 

CO 

Is- 

00 

C0 

CO 

05 

00 

05 

o 

00 

05 

T- 

00 

05 

05 

05 

w 

O) 

05 

05 

05 

05 

.id 

O 

0 

05 

o 

o 

05 

o 

05 

05 

05 

o 

o 

x: 

d 

o 

o 

CO 

05 

in 

05 

05 

m 

05 

05 

CM 

05 

00 

o 

1 — 

in 

CO 

CM 

o 

T— 

T— 

T— 

o 

CM 

CO 

o 

o 

o 

T— 

o 

o 

o 

o 

o 

o 

o 

o 

d 

05 

CO 

05 

in 

05 

05 

05 

CM 

05 

Is- 

05 

T— 

CO 

CM 

o 

O 

in 

00 

in 

CO 

CM 

o 

o 

CM 

T“" 

T” 

o 

o 

o 

d 

d 

o 

o 

o 

o 

00 

CO 

05 

in 

05 

CO 

CO 

05 

O) 

o 

05 

CO 

05 

Is- 

1 — 

CM 

o 

in 

05 

05 

05 

T— 

T"~ 

O 

Is- 

CM 

O 

CO 

T— 

t— 

o 

O 

I— 

T— 

o 

o 

o 

d 

d 

o 

h- 

Is- 

CO 

in 

00 

s 

00 

o 

o 

CD 

05 

ri 

05 

T- 

CO 

i — 

Vw/ 

O 

T— 

CO 

o 

o 

o 

1— 

o 

T— 

▼“ 

1_ ] 

d 

o 

o 

CO 

CM 

CM 

in 

oo 

CO 

r- 

CO 

o 

05 

Is- 

T- 

CO 

r- 

CO 

00 

o 

T— 

o 

CM 

05 

05 

o 

o 

CM 

d 

o 

O 

d 

d 

o 

in 

CD 

1^ 

m 

in 

00 

in 

Is- 

5 

00 

CO 

00 

l- 

05 

Is- 

fc 

o 

Y“* 

CO 

i — 

05 

05 

o 

o 

l— 

o 

O 

O 

1_ ! 

o 

o 

O 

CM 

05 

in 

1^ 

05 

in 

m 

00 

00 

CO 

Is- 

CM 

o 

CO 

CO 

m 

CM 

o 

o 

o 

o 

O 

d 

o 

t- 

d 

T“* 

T_l 

o 

d 

o 

CO 

Is- 

00 

in 

CM 

05 

i^ 

00 

00 

Is- 

T“ 

Is- 

CO 

05 

co 

CO 

00 

00 

CO 

o 

o 

o 

00 

o 

1“ 

5 

o 

d 

o 

o 

o 

o 

o 

CM 

Is- 

00 

,_ 

in 

05 

in 

00 

00 

r- 

Is- 

o 

Is- 

Is- 

CO 

o 

o 

00 

00 

CM 

o 

o 

o 

o 

CM 

Is- 

CM 

d 

d 

o 

o 

o 

r- 

r- 

00 

05 

in 

00 

00 

h- 

CM 

in 

in 

CO 

CO 

O 

Is- 

Is- 

o 

CO 

o 

O 

o 

o 

CO 

o 

o 

o 

d 

o 

d  o 


T“ 

CM 

CO 

m 

CO 

o 

E 

E 

E 

E 

E 

E 

CO 

CO 

CO 

CO 

CO 

0 

0 

0 

0 

0 

0 

0 

1- 

1- 

H* 

1- 

H* 

co 

o 

<o 

o 

£ 

o 

o 

o 


c 

o 

o 


£ 

o 

o 


</> 


to 

to 


Team  3  roll-up  evaluations  prior  to  Competition 


F 

Is- 

T“ 

Is- 

Is- 

Is- 

CD 

F 

T— 

05 

Is- 

£ 

05 

CO 

in 

w 

05 

05 

05 

05 

00 

05 

05 

05 

05 

05 

05 

CD 

w 

05 

05 

05 

05 

05 

05 

— J 

to 

05 

05 

05 

05 

0 

05 

05 

05 

05 

05 

05 

05 

05 

05 

05 

05 

05 

05 

05 

05 

o 

05 

05 

05 

05 

05 

Q 

d 

d 

d 

d 

JsC 

o 

d 

d 

05 

o 

d 

d 

o 

o 

o 

CD 

CD 

o 

x: 

£ 

xz 

o 

O 

o 

o 

€0 

00 

05 

00 

00 

CM 

o 

o 

o 

o 

o 

o 

o 

© 

O 

O 

in 

05 

05 

05 

05 

CM 

CD 

T“ 

CD 

O 

O 

CM 

o 

O 

05 

CD 

d 

o 

CM 

o 

d 

00 

o 

o 

o 

O 

d 

d 

m 

CM 

CM 

05 

CM 

CM 

05 

o 

00 

CO 

o 

o 

o 

05 

O 

O 

m 

05 

00 

CD 

05 

CO 

CD 

O 

o 

O 

o 

o 

& 

Is- 

CD 

o 

o 

CM 

CM 

d 

d 

CD 

O 

o 

d 

d 

o 

d 

o 

d 

00 

CD 

05 

05 

o 

05 

CM 

CO 

T- 

CD 

CO 

CD 

o 

00 

CO 

CO 

O 

in 

05 

05 

05 

05 

CD 

o 

05 

m 

O 

o 

o 

CD 

O 

o 

o 

o 

in 

in 

CM 

CO 

S 

CO 

05 

r* 

o 

CM 

CM 

o 

05 

o 

O 

o 

d 

d 

00 

CO 

o 

d 

d 

d 

o 

r- 

CD 

CM 

CD 

05 

05 

N 

V“ 

00 

CO 

CD 

05 

00 

h* 

CD 

in 

CM 

h- 

T — ■ 

CO 

in 

05 

X 

05 

m 

CO 

05 

05 

05 

CM 

i — 

CO 

Q 

o 

in 

CM 

W 

CO 

CD 

o 

Is- 

ID 

T— 

05 

o 

d 

d 

d 

Is- 

d 

CM 

d 

*— 

LO 

d 

d 

d 

d 

d 

d 

o 

d 

d 

<0 

05 

t: 

00 

CD 

05 

CM 

CD 

■  ^ 

05 

CM 

CO 

05 

00 

CD 

CO 

CD 

CO 

05 

CO 

o 

CO 

05 

CO 

CM 

O 

CM 

Is- 

05 

05 

CD 

T— 

05 

ID 

CM 

CM 

d 

d 

t— 

d 

T““ 

CD 

CM 

d 

d 

d 

CM 

d 

o 

o 

CO 

CO 

Is- 

CM 

T~ 

in 

o 

d 

d 

o 

o 

d 

d 

d 

Q 

10 

05 

T— 

o 

CD 

CD 

00 

O 

in 

CM 

CD 

CM 

CO 

05 

CO 

ft 

m 

CO 

05 

05 

05 

to 

T“ 

CM 

o 

X 

O 

CO 

T — 

05 

05 

w 

05 

05 

t — 

T — 

CM 

CO 

X 

T 

w 

T“ 

T“ 

v— 

h- 

V 

o 

'M' 

CO 

T— 

05 

d 

d 

d 

W 

T“* 

w 

d 

d 

o 

■  T* 

CM 

O 

CO 

0 

0 

d 

d 

0 

d 

o 

d 

c 

O 

C 

c 

E 

** 

05 

CM 

o 

CD 

00 

o 

£ 

CD 

o 

CM 

05 

CD 

o 

£ 

*3* 

CO 

in 

05 

05 

CO 

05 

o 

CO 

Is- 

o 

O 

■'sf 

05 

O 

CM 

o 

d 

CM 

d 

d 

d 

o 

d 

d 

d 

05 

CM 

CO 

Is- 

CO 

CM 

d 

d 

O 

o 

CO 

CM 

o 

o 

05 

o 

o 

IT 

CO 

CM 

h- 

o 

h- 

o 

o 

CO 

CD 

in 

05 

v- 

05 

o 

T — 

O 

o 

05 

CD 

CM 

o 

d 

h- 

V 

o 

V) 

d 

d 

d 

£ 

ft 

T — 

o 

Is- 

CM 

05 

:=s 

w 

Q 

d 

d 

o 

o 

0 

CM 

CO 

o 

o 

05 

o 

o 

c 

8 

CM 

o 

CO 

o 

CM 

o 

o 

1 

CM 

CM 

00 

o 

05 

o 

CO 

o 

CO 

o 

d 

Is-* 

CM 

O 

I 

B 

d 

d 

c 

o 

o 

f 

d 

CO 

CO 

d 

o 

d 

d 

c 

05 

o 

c 

v— 

o 

o 

o 

05 

o 

o 

a. 

o 

o 

o 

00 

o 

o 

v— 

o 

CO 

o 

o 

05 

> 

v— 

ft 

o 

c 

05 

O 

1 

d 

g 

d 

o 

u. 

d 

T- 

CM 

m 

CO 

O 

o 

o 

o 

o 

T— 

CM 

CO 

in 

CD 

T — 

V— 

CM 

CO 

in 

CD 

v— 

v— 

CM 

CO 

£ 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

£ 

£ 

tO 

CO 

m 

to 

m 

to 

to 

to 

to 

to 

to 

to 

to 

CO 

CO 

0) 

CD 

CD 

CD 

CD 

CD 

05 

CD 

CD 

CD 

CD 

CD 

<D 

05 

05 

H 

H 

H 

H 

H 

H 

H 

H 

H 

h- 

h- 

H 

h- 

1- 

H 

Team  5  0.2233  0.5283  0.0733  0.045  0.055  0.06  0.015  0  0  0  0.9999 

Team  6  0  0  0  0.48305  0.37305  0.14305  0  0  0  0  0.99915 

Team  10  0  0  0  0  0  0.31985  0.41985  0.19985  0.06  0  0.99955 


S  (score)  Bins  1-10 

123456789  lOchecksum 

Teaml  0  0.084  0.41845  0.26345  0.14345  0.069  0.021  0  0  0  0.99935 

Team  2  0  0  0.1384  0.4657  0.1808  0.0445  0.0736  0.0567  0.0366  0.0036  0.9999 

Team  3  0  0  0.0084  0.0354  0.0524  0.0554  0.0394  0.208  0.441  0.16  1 


t-  in 

■r- 

c 

in 

T- 

in 

T— 

r-  r- 

CM 

1— 

CD 

CD 

o> 

w 

CD 

CD 

CD 

CD 

CD 

CO 

CO 

CD 

E 

in 

CO 

in 

in 

o 

o 

O 

CD 

CO 

CM 

CD 

i — 

o 

O 

0 

3 

CO 

CD 

CD 

Is- 

CD 

CO 

in 

-C 

CD 

CD 

CD 

CD 

CD 

CD 

o 

CD 

CD 

CD 

CD 

CD 

CD 

o  o 

CD 

o 

o 

T— 

m 

o 

O  T- 

o 

o 

CD 

o 

CD 

CD 

O 

o 

o 

1“ 

d 

CD 

CO 

q 

d 

vW 
-  JZ 

o 

o 

O 

O 

o 

in 

o 

O 

O 

O 

O 

o 

T— 

in 

O 

O 

CD 

o 

CM 

in 

O 

o 

in 

O 

in 

o 

d 

CD 

CO 

00 

CO 

o 

d 

T 

d 

CD 

o 

in 

O 

O 

CM 

o 

CD 

CO 

CD 

o 

CM 

o 

LD 

rt 

o 

O 

0.05 

0.26 

O 

t 

00 

o 

0.4 

in 

CD 

CM 

co 

O 

o 

0.285 

00 

o 

O 

O 

in 

00 

o 

in 

CO 

o 

CM 

CD 

s 

CD 

c 

o 

00 

O 

00 

CD 

T— 

ffl 

1^ 

o 

CM 

in 

o 

CD 

in 

d 

CD 

q 

in 

00 

o 

O 

o 

CO 

CD 

CO 

O 

o 

d 

O 

CO 

d 

o 

co 

o 

O 

d 

00 

T” 

o 

h* 

CM 

o 

in 

o 

CD 

CO 

h- 

d 

m 

o 

o 

CD 

<D 

CO 

T — 

CD 

CM 

CO 

00 

T — 

CD 

o 

in 

1 — 

CO 

CO 

o 

r- 

in 

o 

T— 

in 

d 

00 

T— 

q 

o 

q 

CD 

tJ- 

T— 

o 

CD 

CD 

q 

CO 

d 

ur 

in 

o 

CO 

o 

CM 

o 

CO 

o 

o 

CO 

o 

CM 

o 

CD 

CD 

o 

in 

d 

in 

o 

o 

if# 

o 

CD 

h- 

CO 

o 

CO 

Ilf 

CO 

CD 

in 

CD 

i — 

00 

U) 

CO 

T*“ 

o 

CM 

h- 

T“ 

m 

m 

in 

in 

o 

in 

o 

o 

o 

o 

O 

CM 

o 

CO 

in 

r- 

CD 

c 

CD 

CX) 

co 

o 

o 

CM 

O 

o 

CD 

o 

u 

o 

O 

o 

q 

CM 

d 

ip 

CO 

q 

d 

w 

Q 

in 

CO 

CO 

LO 

o 

o 

m 

CD 

o 

CM 

o 

CD 

o 

w 

V 

CO 

CM 

Is- 

CO 

vm 

o 

in 

Is- 

o 

CM 

U 

CO 

c 

m 

CM 

o 

CM 

00 

O 

5 

CO 

q 

1.26975 

o 

o 

m 

CD 

i — 

CM 

o 

in 

CO 

o 

o 

CM 

d 

0.31 

o 

o 

o 

in 

o 

CO 

o 

in 

CO 

d 

in 

d 

o 

o 

D) 

O 

o 

o 

in 

00 

CO 

C 

CM 

CD 

m 

1 — 

1 — 

i — 

00 

o 

CD 

CO 

CO 

CM 

1“ 

LO 

T“ 

LO 

o 

CO 

LO 

o 

o 

o 

• 

o 

CO 

CO 

CO 

s 

c 

CD 

CD 

d 

T*~ 

o 

o 

o 

d 

CM 

0 

'M; 

Vj 

CO 

CM 

in 

CO 

CO 

■Ql 

o 

o 

o 

3 

O 

CO 

in 

o 

CO 

o 

*-* 

h- 

"M* 

in 

CM 

CO 

T - 

1 — 

CO 

Is- 

'■w' 

-* 

CM 

O 

CO 

CM 

in 

o 

o 

CM 

0.15 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

o 

d 

CM 

00 

CD 

T— 

o 

o 

CM 

CM 

CM 

q 

CM 

T— 

CD 

Q 

o 

CD 

00 

o 

o 

0 

T— 

T“ 

o 

o 

T— 

o 

o 

CM 

o 

"M- 

o 

a_ 

O 

o 

o 

d 

d 

o 

CO 

O) 

r- 

O 

— 

o 

T— 

o 

o 

75 

c 

CM 

O 

CO 

o 

c 

’Z 

o 

IS 

o 

CM 

u 

CO 

d 

:sa 

2 

■*-» 

o 

o 

c 

o 

in 

CD 

T— 

T“ 

CM 

CO 

LO 

CD 

T- 

o 

T— 

CM 

CO 

LO 

CD 

1 — 

E 

E 

E 

E 

E 

E 

E 

E 

E 

o 

1 

E 

E 

E 

E 

E 

E 

03 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

</) 

0 

0 

0 

0 

0 

0 

1- 

h- 

H 

H 

1- 

H 

H 

o 

h- 

h- 

I— 

h- 

H 

Goals  /  weighting  Complete  goal  rollup  of  Bins  1-10 

Control 


1= 

CD 

CD 

in 

CD 

CM 

00 

w 

CO 

CO 

T— 

h*. 

M 

CO 

-J 

in 

h- 

CD 

CD 

CO 

CD 

a> 

03 

CD 

CD 

CD 

CD 

-id 

cd 

CD 

CD 

CD 

CD 

CD 

O 

CD 

_C 

CD 

CD 

CD 

CD 

CD 

CD 

d 

d 

d 

o 

d 

d 

o 

o 

CO 

<0 

CD 

00 

CO 

CM 

T- 

o> 

CD 

CD 

CD 

CM 

CD 

o 

T~ 

CD 

o 

o 

in 

o 

T— 

CD 

o 

o 

CM 

d 

o 

d 

o 

o 

d 

o 

d 

d 

o> 

CM 

CM 

CO 

CM 

CM 

CD 

'M* 

00 

CD 

O 

CO 

CO 

o 

o 

CD 

o 

CO 

o 

o 

o 

CD 

d 

CO 

o 

o 

CO 

d 

d 

s 

d 

d 

d 

CO 

CO 

CM 

CO 

s 

CD 

CD 

o 

CD 

CD 

00 

i — 

CD 

o 

o 

CO 

in 

O 

CD 

o 

d 

CD 

o 

h- 

o 

d 

CO 

o 

SI 

d 

d 

d 

d 

CD 

CM 

h*. 

in 

CD 

in 

CO 

00 

CO 

in 

CO 

CD 

1 — 

¥ 

CO 

CO 

CD 

'r— 

s 

T— 

in 

o 

CO 

o 

y— 

o 

o 

d 

o 

CO 

d 

d 

o 

CO 

CM 

CD 

CM 

00 

T- 

o 

00 

in 

’M' 

T~ 

LD 

CO 

CO 

00 

CD 

K 

o 

o 

CD 

CO 

CO 

CM 

r- 

o 

o 

CM 

T- 

o 

d 

d 

CO 

o 

d 

o 

o 

m 

CM 

CD 

CD 

00 

CM 

CM 

CM 

in 

00 

00 

00 

CM 

CD 

CM 

CO 

T— 

h- 

CD 

00 

LD 

CM 

00 

CO 

T— 

o 

CM 

in 

T— 

T— 

d 

CM 

o 

d 

d 

d 

d 

d 

CM 

CM 

CD 

o 

K 

00 

CD 

CD 

CD 

CO 

CD 

o 

h- 

CD 

T“ 

00 

CD 

CD 

CD 

CM 

h*. 

o 

CM 

CO 

CD 

d 

CM 

o 

d 

d 

d 

CO 

CO 

in 

CD 

CM 

N. 

o 

CM 

o 

Tl* 

in 

00 

xr 

in 

CO 

h- 

o 

h- 

in 

00 

o 

CO 

s 

h* 

CM 

d 

T— 

o 

o 

d 

d 

o 

CM 

CM 

00 

CM 

o 

o 

cd 

h- 

h- 

CM 

in 

CO 

co 

o 

CM 

o 

CD 

N- 

o 

CD 

W 

o 

o 

^r 

d 

o 

o 

CO 

o 

CM 

o 

o 

CD 

CD 

o 

o 

o 

CM 

d 

h- 

T— 

o 


CM 

00 

in 

CD 

o 

T — 

E 

E 

E 

E 

E 

E 

m 

Cd 

cd 

cd 

m 

cd 

Q> 

CD 

o 

CD 

CD 

CD 

H 

I- 

H 

h- 

H 

H 

CO 

co 

P 


IS 

o 

o 

£ 

o 

CD 

w 

c 

o 

o 

o 

Q 

m 

Team  5  roll-up  evaluations  prior  to  Competition: 


E 

i— 

CO 

T— 

■*— 

1— 

p 

CO 

r- 

1^ 

T— 

CD 

c 

CO 

h- 

in 

CD 

CD 

00 

L. 

CD 

CD 

CD 

CD 

L. 

CD 

CD 

CD 

— > 
(0 

o> 

CD 

CO 

CD 

CD 

CD 

CD 

<0 

CD 

CD 

CD 

CD 

cd 

cd 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

o 

o> 

CD 

o 

d 

O 

O 

O 

o 

O 

vj 

CD 

O 

CD 

d 

o 

0 

0 

O 

x: 

x: 

o 

o 

o 

o 

00 

o 

1 — 

CM 

CD 

o 

T— 

CM 

O 

o 

00 

CO 

o 

CM 

1— 

in 

O 

r- 

a> 

CD 

LO 

00 

CO 

CO 

CD 

CO 

CO 

o 

CO 

r* 

T— 

t — 

CO 

CO 

T“ 

1^. 

CD 

m 

o 

CO 

Q 

o 

d 

T- 

o 

O 

CM 

o 

u 

d 

'  . 

o 

d 

d 

o 

d 

o 

O) 

00 

N. 

v- 

CD 

V— 

CO 

O) 

CM 

CD 

o 

00 

CO 

o> 

CM 

in 

in 

o 

CD 

O 

CD 

CO 

1“ 

CO 

CM 

CD 

CD 

CO 

CO 

CO 

O 

in 

o 

CD 

CD 

CM 

m 

O 

o 

CD 

o 

cd 

o 

CD 

o 

d 

CM 

o 

v— 

CM 

CO 

o 

d 

■ 

o 

o 

d 

d 

o 

o 

00  00 

CD 

CM  o  o 

00  t- 

CM 

CD  O  00 

CO 

00  i- 

moo 

CD 

00 

CD 

T — 

co 

CD  CD 

CO 

00 

CD 

in 

.N- 

CM 

q 

V— 

CO 

i-  in 

CO 

v— 

?! 

d 

in 

o 

1-  CO 

CO 

o 

CM 

d 

q 

o 

o  d 

d 

o 

o 

CM 

CM 

O 

O 

O 

O 

CD 

CM 

CD 

O 

CD 

o 

h- 

CD 

LO 

in 

o 

CO 

CO 

o 

CD 

CD 

CO 

CD 

CD 

CO 

T“ 

o 

i — 

r* 

o 

CD 

CD 

d 

m 

d 

CM 

o 

1 — 

CO 

o 

o 

o 

d 

o 

o 

CO 

CD 

00 

CD 

O 

O 

O 

CO 

o 

CM 

CM 

O 

CD 

o 

CO 

CD 

CO 

LO 

o 

CO 

T— 

CD 

00 

in 

o 

o 

d 

00 

o 

T— 

CD 

CD 

o 

o 

d 

o 

CM 

d 

d 

h- 

o 

o 

o 

m 

o 

CD 

o 

o 

Q 

in 

o 

CM 

CM 

CO 

o 

Q 

in 

CM 

CO 

LO 

CO 

Y— 

■ 

T" 

in 

CO 

o 

CM 

d 

Y“ 

o 

■ 

Y— 

o 

d 

CM 

o 

O 

o 

CD 

o 

d 

w 

1 

r- 

o 

o 

o 

d 

CO 

CO 

o 

o 

CO 

o 

in 

tn 

CO 

c 

c 

c 

o 

5 

o 

CO 

O 

00 

o 

o 

S3 

o 

CO 

CD 

CD 

o 

m 

o 

00 

CM 

00 

m 

^r 

o 

o 

CM 

CD 

CM 

CO 

o 

d 

d 

o 

d 

o 

I— 

'M- 

d 

d 

o 

o 

T“ 

d 

00 

o 

CM 

o 

CD 

o 

o 

£ 

CO 

o 

CM 

o 

CD 

CD 

o 

00 

o 

m 

CM 

CM 

o 

i— • 

o 

O 

w 

o 

m 

CO 

00 

CO 

CO 

d 

d 

d 

o 

o 

o 

2 

o 

CO 

T— 

LO 

o 

LO 

CO 

o 

o 

tn 

d 

d 

d 

1m 

*■» 

CM 

o 

o 

o 

o 

o 

o 

c 

0 

CM 

o 

o 

o 

00 

o 

o 

2 

CM 

o 

CD 

CD 

CM 

0 

i — 

CD 

o 

CO 

t 

</) 

d 

c 

o 

o 

T— 

d 

o 

CM 

CM 

1 

d 

c 

1 

O) 

o 

o 

o 

o 

o 

o 

o 

Q. 

Y“ 

o 

o 

o 

o 

o 

o 

c 

o 

CM 

o 

> 

O 

o 

CM 

CO 

1m 

c 

0 

£ 

o 

d 

d 

CM 

o 

w 

d 

^co 

T" 

CM 

CO 

CO 

o 

o 

o 

o 

o 

V— 

CM 

CO 

m 

CD 

*1— 

T— 

CM 

CO 

LO 

CD 

1— 

1 — 

CM 

CO 

LO 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1- 

1- 

H 

H 

H 

H 

1- 

H 

H 

1- 

H 

1- 

1- 

H 

h“ 

H 

Team  6  0.046  0.051  0.056  0.061  0.101  0.1  0.105  0.1  0.25  0.13 

Team  10  0  0  0  0  0  0  0.03  0.3  0.39  0.28 


S  (score)  Bins  1-10 

1234567  89  lOchecksum 

Teaml  0.13931  0.41331  0.16731  0  0  0  0  0.05399  0.15999  0.06599  0.9999 

Team  2  0.34645  0.22245  0.10445  0.006  0  0.018  0.0555  0.06981  0.10881  0.06781  0.99928 


to 

OJ 

05 

05 

05 

O 


O)  O  r-  ^ 

05  CM  ID 

O  O  v— 

B  o  d 

© 


05  ©  t- 
05  CM  CO 


05 

CD 

CO 

o 

o 

© 


E 

3 

m 

o 

© 

O 

© 


05 


to 

05 

05 

05 

d 


CM  O  O  O  t— 

d  o 


o  o  o 


£ 

co 

05 

to 

CM 

t. 

05 

K 

CM 

h- 

00 

05 

05 

to 

05 

05 

05 

05 

05 

CD 

© 

O 

CD 

.C 

d 

d 

05 

05 

o 

05 

o 

d 

o 

o 

h- 

CD 

LO 

o 

T“ 

T— 

CD 

fc 

CM 

o 

o 

T— 

O 

d 

o 

o 

CD 

o  co  in 

LO 

LO  O  LO 

CD 

o 

05 

~  CM 

CO 

O  CO 

CM 

o 

“  t 

*  O 

o 

d  ^ 

o 

o 

d 

o 

£ 

£ 


CO  o 

05  O 

8  o 


CO  00  N  O 
K  O 
05  O  q 
CM  H  ° 


K 

CO 

CO 

o 


to 

^r 
CO  o 

8  o 

d 


in 

h-  to 

in 

CD 

CD 

o 

CM 

co 

CM 

o 

d 

d 

o 

d 

o 

d 

m 

CD 

3 

05 


lO  i- 
CO  rfr 

9  d 

o 


in  in  co  o  in  co 

O  CO  CM  CO  y- 

d  T  d  T  o 

o  o 


CD 

to  o 

in 

o  to 

CD 

to 

to 

O 

*** 

£ 

00 

CM 

CO 

00 

O 

o 

O 

T— 

o 

o 

i — 

d 

o 

ts 

o 

o 

d 

© 

3 

cr 


CM 

00 

CM 

d 


O  M-  M 
05  ’M’ 
^  CM 

d  o 


00 

o 

o 

to 

Y“* 

in 

T— 

CD 

1 — 

00 

o 

CM 

o 

o 

d 

05 

T— 

o 

05 

d 

C 

‘5 

o 

o 

CO 

o 

to 

CD 

to 

CD 

O 

CO 

£ 

CD 

CD 

d 

T— 

d 

K 

to 

Y“ 

CM 

h* 

h- 

s 

O 

05 

n 

CM 

O 

o 

o 

o 

d 

CD 

to 

h- 

CD 

CM 

05 

h*. 

05 

in 

CO 

co 

o 

T— 

o 

d 

o 

CO 

to 

in 

05 

k 

co 

h- 

05 

05 

to 

Xt 

CO 

T— 

T“ 

05 

o 

o 

O 

d 

CO 

h- 

in 

'M- 

T —  - 

CD 

CM 

05 

T“ 

in 

Y- 

o 

o 

T~i 

o 

M- 

LO 

00 

o 

O 

N- 

CM 

d 

O 

d 

in 

o 

CO 

d 

o 

03 

r- 

CM 

o 

00 

i — 

d 

o 

3 

to 


to  03  O 
'i-  N- 

d  ^ 
o 


CM 


©  T~ 

o 

o 


© 

£ 


o  to  o  to  o  o 

CD  00 

<0  **- 

d 


CM 

d 


O  O  O  CD  O  O 

to 


05 

£ 

w 

O 

o 

05 


CO 

o 

in 

05 

CM 

CM 

CM 

h- 

LO 

O 

CO 

T— 

o 

^r 

o 

d 

o 

CM 

o 

CM 

CO 

CD 

CD 

CNJ 

T — 

o 

o 

Y“ 

d 

d 

T— 

CM 

d 

Y— 

o 

00 

CM 

CM 

o 

Y“ 

05 

d 

d 

d 

CO 

ID 

CD 

o 

T— 

CM 

CO 

to 

CD 

o 

£ 

o 

T- 

CM 

CO 

in 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

t> 

1 

0) 

E 

E 

E 

E 

co 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

<d 

CO 

CO 

CO 

© 

05 

05 

05 

05 

05 

05 

05 

05 

05 

© 

© 

© 

© 

© 

1- 

H 

H 

1— 

H 

H 

H 

H 

H 

h- 

o 

H 

H 

H 

H 

Team  6  0.0322  0.0357  0.05  0.0625  0.0995  0.0958  0.0933  0.17794  0.23794  0.11494  0.99982 

Team  10  0  0  0  0  0  0  0.021  0.24999  0.45399  0.27499  0.99997 


Goals  /  weighting  Complete  goal  rollup  Bins  1-10 

Control 


£ 

00 

T— 

CO 

CO 

u. 

CM 

CO 

CD 

T— 

03 

03 

1^ 

CD 

CO 

CD 

CD 

CD 

CD 

00 

CD 

CD 

03 

CD 

CD 

O) 

CD 

CD 

03 

03 

CD 

CD 

CD 

CD 

CD 

03 

vU 

x: 

d 

d 

d 

O 

O 

d 

o 

o 

CO 

CD 

in 

CM 

CM 

h- 

r- 

CO 

o 

in 

00 

03 

CD 

CO 

1 — 

CD 

00 

in 

S 

in 

q 

in 

CO 

m 

00 

o 

o 

o 

vJ 

o 

CM 

o 

o 

d 

o 

o 

O 

CO 

T~ 

CO 

CO 

CM 

Is- 

1 — 

CD 

CO 

00 

03 

o 

CO 

o 

o 

o 

i— 

CO 

in 

CO 

o 

m 

o 

h- 

CD 

CD 

Is- 

1— 

o 

o 

CO 

d 

o 

o 

o 

d 

00 

CM 

T- 

CO 

CM 

CM 

r- 

CM 

00 

CD 

CD 

in 

^r 

o 

CO 

in 

CM 

m 

q 

h- 

CM 

o 

N- 

T- 

T— 

o 

w 

o 

CO 

d 

d 

d 

d 

d 

1^ 

CD 

in 

r- 

CO 

CD 

CO 

h- 

CO 

CO 

03 

CM 

LO 

00 

Is- 

o 

CD 

CO 

CD 

CO 

o 

CO 

O 

00 

o 

CM 

T— 

CO 

o 

o 

o 

o 

o 

o 

d 

o 

to 

CD 

CD 

CO 

00 

o 

CM 

CM 

s 

o 

Is- 

o 

CO 

o 

in 

CO 

00 

o 

1^ 

CO 

o 

o 

o 

CO 

d 

o 

o 

d 

d 

in 

CM 

Is- 

00 

t: 

in 

o 

CD 

00 

o 

CO 

00 

CM 

CO 

o 

CM 

CO 

O 

in 

o 

O 

o 

o 

CO 

o 

o 

o 

o 

o 

00 

CM 

00 

in 

o 

T— 

CO 

in 

T— 

CO 

CO 

CD 

in 

1 — 

CM 

o 

in 

■ 

o 

o 

CM 

o 

o 

o 

CO 

CO 

m 

CM 

CM 

in 

o 

00 

in 

h- 

h- 

CO 

in 

T“ 

Is- 

q 

o 

in 

— 

CD 

o 

CD 

1 — 

O 

kJ 

v— 

o 

o 

CO 

o 

o 

d 

d 

CM 

CO 

CO 

CD 

CM 

T- 

o 

00 

CO 

00 

Is- 

T— 

CD 

00 

T— 

Is- 

CO 

q 

CD 

in 

1 — 

x 

in 

i — 

CM 

d 

v — > 

CO 

d 

o 

d 

i— 

CO 

N- 

CM 

CD 

o 

00 

CM 

Is- 

CM 

in 

CD 

Is- 

in 

CO 

T— 

q 

03 

00 

CM 

/—v 

CO 

CM 

o 

d 

V. — > 

o 

o 

o 

T— 

CM 

CO 

in 

CO 

o 

T— 

E 

E 

E 

E 

E 

E 

CO 

CO 

(0 

cO 

CO 

CO 

<13 

CD 

CD 

0 

CD 

0 

1- 

h- 

H 

H- 

H 

H- 

co 

(O 

o 

o 

0) 

o 

o 

(0 

o 

k. 

£ 

c 

o 

o 

o 

o 

<0 

c 

CM 

CO 

T— 

f— 

T~ 

P 

K 

y— 

CD 

T~ 

Y— 

Y— 

P 

ID 

CO 

ID 

T- 

T— 

T — 

—1 

o 

CD 

o 

w 
•— * 

CD 

CD 

— * 

CD 

o 

—J 

(f) 

CD 

CD 

CD 

— > 

m 

CD 

CD 

CO 

CD 

CD 

05 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

o 

d 

CD 

CD 

o 

d 

d 

o 

CD 

o 

CD 

0 

d 

d 

o 

0 

d 

d 

-C 

.c 

-C 

o 

o 

Q 

o 

00 

CD 

T— 

CM 

t— 

CD 

o 

O 

o 

o 

CM 

CO 

o 

o 

ID 

ID 

o 

CO 

CM 

T— 

N. 

CD 

o 

ID 

CO 

T“ 

T“ 

f* 

CO 

05 

X 

O 

o 

o 

CO 

3 

00 

T — 

CM 

- 

d 

d 

o 

d 

d 

O 

o 

CD 

S 

w 

d 

o 

d 

d 

m 

CD 

-r- 

CD 

o 

CO 

0) 

V* 

tj- 

CO 

o 

CD 

ID 

CD 

CD 

ID 

ID 

o 

CO 

CD 

CM 

CD 

o 

CO 

o 

CM 

CO 

00 

’M’ 

CD 

CM 

o 

o 

CM 

CM 

d 

CO 

CD 

O 

CM 

CO 

CO 

o 

d 

o 

d 

d 

ID 

O 

d 

T“ 

o 

o 

CD 

O 

O 

O 

o 

CO 

CO 

o 

o 

d 

d 

o 

o 

€0 

CD 

1— 

CM 

o 

CD 

€0 

CM 

CM 

CM 

o 

CM 

00 

o 

ID 

ID 

CM 

CM 

CM 

CD 

o 

o 

o 

CO 

00 

CD 

CM 

ID 

o 

N- 

CM 

d 

r* 

CO 

o 

CD 

5 

o 

d 

o 

d 

o 

h- 

o 

d 

CM 

d 

d 

ID 

o 

CO 

CM 

o 

o 

d 

O 

d 

d 

o 

o 

d 

h. 

o 

o 

CD 

o 

o 

1"- 

CO 

o 

CM 

o 

CM 

o 

h- 

'sf 

ID 

CO 

ID 

CO 

€0 

p 

CO 

CO 

CD 

O 

CO 

CO 

CD 

CM 

CM 

O 

X 

CO 

T — 

LD 

T — 

t — 

T — 

T — 

o 

d 

w 

CM 

d 

CO 

d 

o 

o 

d 

d 

o 

d 

d 

d 

d 

CD 

O 

O 

O 

CD 

o 

o 

CD 

O 

05 

o 

Y“  • 

O 

CD 

ID 

ID 

05 

ID 

CM 

CD 

p 

CD 

/X 

00 

CD 

05 

CM 

h- 

xj- 

O 

o 

CO 

w 

CD 

O 

ID 

CM 

T“ 

O 

d 

w 

CO 

O 

d 

y“ 

O 

d 

d 

CM 

VD 

d 

d 

d 

d 

a 

ID 

O 

O 

o 

CD 

o 

o 

o 

ID 

CD 

t— 

CD 

00 

o 

ID 

ID 

CD 

05 

ID 

w 

h- 

w 

05 

CM 

CM 

CD 

00 

w 

CD 

o 

05 

r- 

ID 

ID 

d 

T 

T- 

ID 

T — 

d 

d 

T“ 

d 

o 

d 

CD 

O 

ft 

d 

05 

O 

h- 

o 

Y“ 

d 

O 

o 

0 

w 

o 

0 

w 

d 

d 

c 

■  £ 

£ 

d 

15 

o 

o 

o 

CD 

o 

o 

s 

CD 

CD 

CD 

CD 

-M- 

o 

s 

LO 

CM 

T— 

ID 

05 

ID 

05 

O 

O 

00 

ID 

CD 

CO 

CO 

ID 

'M* 

H 

© 

CM 

d 

05 

O 

d 

d 

CM 

d 

o 

d 

CD 

T“ 

ft 

N- 

O 

ID 

CM 

T“ 

o 

d 

o 

d 

g 

d 

W 

d 

d 

d 

d 

^£3 

0 

& 

CO 

o 

CD 

o 

00 

o 

o 

£ 

CO 

05 

o 

o 

CD 

CO 

o 

CO 

ID 

T- 

CM 

ID 

CM 

Tt 

g 

05 

CD 

0 

CD 

CD 

p 

CD 

CO 

CO 

CM 

CO 

f— 

w 

O 

CO 

CO 

0 

CD 

h*. 

CO 

O 

o 

O 

© 

u 

o 

o 

d 

CO 

d 

d 

w 

u 

o 

o 

CM 

O 

d 

T“ 

d 

CM 

d 

d 

d 

© 

d 

o 

0 

d 

1m* 

4-* 

CM 

o 

05 

o 

CO 

o 

o 

£ 

ft 

CM 

CO 

o 

o 

CD 

CD 

o 

o 

3k. 

CM 

o 

CM 

r* 

CO 

o 

W 

05 

00 

u 

#% 

x 

O 

CO 

CO 

ft 

CM 

05 

o 

w 

w 

y — ■ 

o 

£ 

T — 

ID 

w 

O 

& 

50 

o 

o 

d 

0 

d 

d 

O 

o 

CO 

d 

o 

d 

d 

S3 

d 

£ 

O? 

J 

#© 

o 

c 

r- 

o 

CD 

o 

o 

o 

o 

Q. 

o 

o 

o 

CD 

o 

o 

Y— 

o 

05 

o 

CD 

ID 

© 

o 

05 

>* 

CO 

ft 

05 

CO 

O 

x 

i 

*3 

V- 

i*-f 

C 

o 

ID 

O 

1 

o 

d 

w 

o 

^0 

LD 

d 

’M- 

d 

o 

d 

w 

> 

o 

d 

T** 

CM 

0 

CO 

m 

a 

© 

O 

o 

o 

a 

*© 

v© 

o 

o 

o 

T- 

CM 

CO 

ID 

CD 

T“ 

T — 

CM 

CO 

ID 

CD 

r— 

T— 

CM 

CO 

ID 

CD 

1 — 

B 

m 

E 

E 

E 

E 

E 

£ 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

© 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H* 

H 

H 

H 

i— 

H 

H 

H 

H 

S  (score)  Bins  1-10 

123456789  lOchecksum 

Team  1  0  0.03731  0.02331  0.00931  0.26  0.195  0.175  0.2  0.07  0.03  0.99993 

Team  2  0.15  0.48731  0.17331  0.00931  0  0.006  0.018  0.02595  0.04995  0.07995  0.99978 


00 

T— 

-T— 

T— 

to 

to 

to 

to 

1— 

1— 

T" 

Is- 

1^ 

Is- 

Is- 

cd 

— J 

w 

CD 

o> 

o> 

cd 

CD 

CD 

cd 

CD 

CD 

to 

to 

cd 

O) 

cd 

cd 

CD 

E 

1 — 

T— 

Y- 

T— 

o 

o 

© 

o 

o 

d 

d 

3 

« 

CO 

to 

CD 

T- 

CM 

sz 

O) 

CD 

CD 

o 

CD 

o> 

CD 

T— 

o 

00 

T— 

o 

o 

o 

o 

o 

o 

to 

o 

CD 

X 

CD 

cm 

q 

d 

CO 

o 

CM 

© 

JZ 

o 

o 

vj 

d 

d 

d 

o 

o 

to 

to 

o 

CO 

3 

T" 

CO 

CO 

to 

CD 

CM 

cd 

o 

o 

o> 

o> 

o 

o 

o 

o 

to 

o 

o 

£j 

d 

o 

CO 

o 

CO 

o 

Is- 

o 

o 

o 

d 

d 

O) 

CO 

to 

to 

o 

CD 

fc 

CD 

CM 

CO 

Is- 

Y- 

00 

CM 

to 

Y— 

to 

o 

CD 

o 

— • 

to 

o 

o 

CO 

o 

00 

o 

o 

o 

o 

to 

CM 

to 

CM 

o 

o 

o 

o 

o 

o 

co 

O 

V 

o 

CM 

d 

Is- 

T“ 

CA 

CM 

o 

CO 

00 

CO 

to 

to 

CO 

to 

X 

o 

^st* 

o 

CO 

q 

w 

C 

to 

o 

CM 

CO 

CM 

IO 

CO 

CM 

o 

m 

N 

o 

to 

CO 

o 

to 

CM 

CM 

CO 

o 

00 

o 

o 

o 

d 

o 

CM 

CM 

o 

II 

d 

CD 

o 

o 

CO 

o 

to 

o 

o 

Si 

d 

d 

q 

Is- 

00 

to 

Is- 

to 

00 

d 

d 

o 

s 

f2 

CM 

CD 

s 

CO 

to 

o 

Is- 

o 

o 

in 

CM 

o 

CO 

to 

to 

00 

o 

CO 

to 

CM 

o 

q 

o 

d 

o 

cd 

CM 

CM 

CM 

CM 

o 

o 

d 

d 

co 

"sT 

q 

q 

CO 

CO 

CO 

d 

d 

d 

CO 

o 

U) 

o 

CM 

CO 

to 

to 

CD 

to 

00 

o 

W 

© 

3 

o 

d 

to 

00 

to 

CO 

Y— 

Is- 

lO 

o 

to 

o 

CO 

o 

CM 

O) 

IO 

to 

to 

to 

to 

to 

o 

to 

o 

o 

o 

CM 

o 

q 

o 

o 

o 

CO 

1 — 

c 

Is- 

CM 

CM 

CM 

to 

d 

d 

T_ 

o 

o 

o 

o 

CO 

CO 

CO 

CM 

q 

ri 

o 

d 

*3 

(0 

o 

CO 

CO 

CO 

CM 

CO 

o 

d 

o 

IO 

to 

to 

CO 

to 

1 — 

00 

© 

3 

o 

d 

d 

o 

1 

CM 

CO 

o 

T“ 

CD 

o 

CM 

CO 

5 

o 

CM 

i — 

tr 

to 

CO 

to 

to 

CO 

o 

© 

CM 

to 

o 

o 

5 

o 

Y — 

T“* 

o 

o 

o 

to 

CM 

CM 

Is- 

CM 

CM 

CM 

CM 

c 

d 

d 

q 

Q 

o 

d 

O 

d 

© 

to 

Is- 

CM 

d 

CO 

CO 

Y — 

CO 

CO 

CM 

d 

5 

d 

to 

Is- 

to 

to 

00 

O) 

* 

CM 

CM 

00 

Is- 

o 

c 

o 

o 

o 

CO 

00 

LO 

to 

CO 

w 

1 — 

Is- 

CD 

00 

o 

o 

CM 

00 

CO 

to 

00 

to 

to 

to 

o 

o 

q 

'T™. 

T“ 

o 

o 

CO 

CM 

T“ 

CM 

Is- 

q 

CM 

CM 

to 

d 

o 

d 

o 

CO 

o 

O 

c 

co 

Ok 

CO 

CO 

o 

d 

to 

o 

CM 

o 

d 

d 

© 

"D 

CO 

CO 

CO 

d 

CO 

to 

Is- 

^r 

to 

00 

d 

3 

VJ 

o 

o 

CM 

CO 

T“ 

CM 

CO 

CD 

CM 

00 

CO 

CO 

© 

o 

CM 

CD 

to 

o 

o 

Is- 

o 

Is- 

s*’^' 

CM 

LO 

o 

o 

IO 

o 

o 

CM 

’“I 

q 

CM 

d 

d 

CO 

o 

CM 

CM 

T. 

o 

d 

d 

CD 

CD 

o 

CD 

o 

to 

o 

o 

o 

CM 

CD 

Is- 

00 

CD 

o 

o 

v — > 

o 

q 

1 — 

CM 

CO 

Is- 

CD 

r- 

Y“* 

Y— 

Y — 

co 

Y — 

o 

5 

C\J 

CO 

Is- 

Is- 

LO 

0.72 

1 — 

d 

© 

a. 

O 

o 

CO 

o 

o 

o 

o 

soo 

o 

o 

CM 

d 

o 

d 

d 

d 

o 

O 

© 

O) 

o 

CO 

o 

CO 

to 

CD 

d 

75 

c 

*3 

c 

CD 

Y — 

CO 

k 

o 

o 

CO 

d 

o 

o 

o 

d 

© 

d 

o 

k. 

CO 

to 

CO 

o 

o 

4- 

c 

o 

T— 

T— 

CM 

CO 

to 

CO 

T— 

o 

T— 

CM 

CO 

to 

CO 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

o 

1 

E 

E 

E 

E 

E 

E 

© 

CO 

co 

CO 

© 

© 

© 

© 

© 

© 

1 

c n 

© 

© 

© 

© 

© 

© 

CD 

© 

a> 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

1- 

h- 

H 

l- 

1- 

h- 

H 

1- 

1“ 

h- 

o 

h- 

1- 

f— 

1- 

1- 

Goals  /  weighting  Complete  goal  rollup  Bins  1-10 

Control 


c 

CO 

in 

CD 

T“ 

T— 

c 

h- 

T — 

h* 

CD 

N- 

T— 

CD 

05 

05 

CD 

0) 

CD 

CD 

O 

05 

0> 

CD 

w 

SZ 

O 

d 

d 

o 

o 

CO 

CD 

t— 

CM 

00 

05 

to 

T — - 

CD 

in 

T— 

Tf 

CM 

CO 

CD 

00 

CD 

O 

CD 

O 

o 

CM 

CM 

O 

h- 

rf 

O 

d 

d 

o 

d 

d 

05 

a> 

CD 

r* 

CD 

N- 

00 

CO 

CO 

in 

co 

T — 

CO 

o 

K 

CO 

o> 

05 

o 

t— 

CO 

h- 

o 

CD 

O 

00 

O 

o 

o 

d 

o 

d 

o 

d 

€0 

CD 

CM 

in 

00 

05 

€0 

CD 

o 

00 

in 

CM 

T“* 

CD 

■*— 

o 

CO 

CM 

CO 

rs 

CM 

00 

O 

O 

w 

CM 

o 

T“ 

o 

d 

O 

d 

o 

d 

I-* 

in 

T- 

in 

T — 

00 

00 

in 

00 

CD 

o 

dr~ 

CM 

CD 

o 

05 

in 

in 

CO 

00 

in 

o 

CM 

o 

h- 

o 

o 

o 

CM 

d 

o 

d 

d 

o 

o 

d 

<0 

m 

in 

in 

CM 

'M- 

CD 

CM 

o 

CD 

CO 

CM 

in 

IS 

CM 

05 

00 

05 

T“ 

in 

in 

O 

00 

o 

CM 

CM 

o 

o 

V* 

d 

d 

o 

d 

d 

o 

d 

ID 

00 

in 

05 

in 

CO 

'M* 

00 

i — 

s 

in 

00 

00 

N- 

CO 

in 

^r 

€0 

T“ 

t~ 

o 

CM 

s- 

in 

CM 

h- 

in 

o 

O 

w 

o 

o 

d 

d 

d 

d 

d 

00 

T— 

in 

in 

f- 

in 

in 

05 

00 

o 

o 

in 

CM 

o 

1— 

00 

CD 

00 

00 

CO 

CM 

■*— 

CM 

in 

o 

o 

O 

o 

o 

o 

o 

d 

d 

d 

d 

d 

CO 

CD 

00 

in 

CM 

h- 

CO 

CO 

CM 

S* 

00 

h*. 

CO 

T_ 

00 

T— 

o 

o 

00 

o 

CM 

in 

in 

in 

o 

o 

o 

o 

CM 

d 

o 

d 

d 

d 

d 

CM 

CD 

CD 

00 

CD 

h* 

CM 

CO 

00 

CO 

h" 

CO 

CO 

T — 

r- 

05 

in 

o 

0) 

CD 

in 

CO 

o 

q 

M 

in 

CO 

o 

O 

CO 

o 

d 

d 

w 

d 

d 

d 

o 

00 

CD 

CM 

in 

s- 

h- 

CD 

o 

o 

in 

CD 

CO 

CM 

LO 

h- 

00 

CO 

d 

CM 

CO 

T” 

o 

d 

d 

o 

d 

o 

T— 

CM 

CO 

m 

CD 

T — 

E 

E 

E 

E 

E 

E 

Cd 

CO 

cd 

id 

<0 

eo 

Q> 

CD 

0 

0 

0 

CD 

H 

H 

1- 

H 

1- 

H 

co 

<0 

P 


Z 

O 

o 

m 

o 

© 

c 

o 

o 

o 

o 

CO 

Team  10  roll-up  evaluations  prior  to  Competition: 


CD 

CD 

CD 

O) 


CD 
00 
CD 
CD 
CD 

O  O 


W  S 

g  d 

-C 

o 

©  0  0  0  0  0-3- 

T-  1— 

o 

o 


O)  O)  O  N  CO 

CD  CM  CO 

CD  O  O 

O  O  O 

P  do 

o 


CO  O)  00  CD  CD 
CD  O  CO  CD 
CM  ^  O  O 
°  CM  O 

o 

o 


Is- 

o 

o 


CD 

CM 

o 


o  o 


n  o)  in  o)  o>  in  co 


co  n  o 
oo  -i-  in  i- 

00  f-J  T—  T- 

CM  °  CM 

o  d 


co  CM 
T-  O 

Q  h 


o  o 


(O  in  O)  CM  CM  CM 


in  in  cm  cm 

r>  CM  CD  CD 

o 


CM 

o 


CM 

d 


<o 

c 

5 


in  in  cd  cm 
CM  N-  o  CO 
-3" 

o 


-3- 

d 


m 

m 

CO 

o 


o 

d 


o  m  CD  CM  CM 
co  o 
o  oo 

o  i- 


o 

d 


o  o 


o 

CM 

CM 

o 


CO  o  o 


CM 


N  CD  CD 
CM  CD  O 
O  CD  O 

P  P  d 
o  o 


0  0  0  0  0-3- 
CD 

o 


o 

o 


1-  0  0  0  0  0-3- 

o 
o 


C  -3- 
5  CD 

q 

o  o 
0 
-C 

o 

o  o 


o>  o 


00  o 


CD 

CD 

o> 

CD 

d 


c  in  i-  in  in 

§  in  CD  CD  CD 

55  05  CD  CD  CD 

r;  CD  O)  CD  CD 

o  °i  o  o  o 

CD  O 


CD 

CD 


CO  00 
CD 
CO 
CM 

o 


h- 

co 

o 

o 

1^ 

o 

o 

CO 

in 

O 

O 

co 

-3* 

TT“ 

O 

CD 

o 

co 

o 

o 

o 

d 

CO 

o 

o 

1- 

CO 

o 

o 

Is- 

o> 

o 

CM 

m 

O 

co 

CO 

in 

00 

CO 

d 

00 

i— 

00 

CD 

l- 

o 

T* 

i — 

d 

d 

o 

o 

CO 

00 

-3" 

Is- 

00 

o 

CM 

in 

in 

CO 

CM 

00 

CD 

CO 

CD 

o 

o 

00 

o 

o 

i — 

00 

CD 

o 

o 

d 

o 

o 

CM 

o 

o 

d 

in 

in 

00 

in 

Is- 

r^ 

o 

CM 

o 

CO 

o 

T“ 

00 

CM 

in 

00 

CO 

d 

d 

■»— 

d 

Is- 

o 

o 

o 

o 

o 

o 

in 

00 

00 

CM 

CD 

in 

in 

in 

in 

r* 

00 

CM 

Is- 

00 

-3* 

00 

CO 

Q 

CM 

CM 

o 

CD 

Is- 

-3- 

CD 

o 

o 

o 

q 

rvi 

O 

-3- 

Q 

in 

00 

o 

CD 

00 

CD 

CD 

in 

in 

in 

O 

CD 

CD 

00 

o 

o 

w 

00 

in 

CM 

V 

in 

CD 

co 

CM 

rj 

l 

CD 

CM 

o 

1— 

co 

o 

d 

o 

T— 

CM 

CO 

o 

d 

</> 

CM 

o 

c 

c 

d 

in 

in 

-3- 

CM 

o 

CD 

00 

00 

CD 

in 

1— 

lO 

i — 

CD 

o 

Is- 

P 

00 

CD 

CM 

d 

CD 

T— 

1— 

o 

CD 

O 

d 

o 

d 

d 

o 

co 

d 

-t 

CO 

CO 


in 

CD 

CD 

O 

o 


in  co 
■  r-. 


-3- 

CD 

CM 

O 


CM 

d 


-3- 

£ 

CO  00  O  CD  O  -3- 

CO 

CO 

O  CD  LO 

o 

CO 

o 

O  g  P 

o 

d 

P  o 

o 

CO  1— 
^  CM 

£  O 

o 

o 

</> 


in  m  oo 

CD  o  oo 

in  i-  cm 
Pod 
o 


c 

o 

o 

I 

I 

CO 


o 

a 

> 

I 


</) 

o 


CM  O  O  O  O  O  CO 

2  CM 

m 

CM  O  LO 

o 

4-» 

CD 

CM 

CM 

d 

c 

o 

o 

o 

d 

O 

d 

o 

CM 

o  o  o  o  O  CD 
o 
d 


o> 

c 

i-  in 
O  CM 

o  o 

e  d 

CM 

</> 

o 


o  o  o 


T- 

CM 

CO 

in 

CD 

o 

V— 

CM 

CO 

in 

CD 

O 

CM 

CO 

in 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

co 

CO 

co 

co 

CO 

co 

CO 

co 

CO 

0 

0 

0 

0 

0 

0 

0 

Cl) 

CD 

CD 

CD 

CD 

CD 

CD 

0 

0 

0 

0 

0 

0 

0 

0 

0 

h- 

H 

H 

H 

H 

1- 

1- 

H 

h- 

h- 

H 

1" 

H 

H 

1- 

Team  6  0  0  0.0999  0.1599  0.2499  0.08  0.1432  0.1332  0.1332  0  0.9993 

Team  10  0.53  0.048  0.063  0.078  0.063  0.048  0.03  0.03665  0.04665  0.05665  0.99995 


S  (score)  Bins  1-10 

12345678  9  lOchecksum 

Team  1  0.65  0  0.05661  0.0666  0.0816  0.06999  0.045  0.03  0  0  0.9998 

Team  2  0.137  0.137  0.137  0.137  0.143  0.025  0.0463  0.0703  0.1303  0.037  0.9999 


CO 

CO 

CO 

Y~ 

£ 

h- 

Y— 

Y— 

Y— 

r- 

Y— 

CD 

00 

00 

— * 

CD 

CD 

a> 

05 

05 

05 

CD 

CD 

05 

05 

05 

05 

CD 

ID 

CO 

ID 

ID 

05 

05 

p 

05 

05 

E 

h*. 

Y~ 

O 

a> 

p 

o 

CO 

ID 

CD 

ID 

CD 

O 

o 

o 

0 

w 

0 

ID 

05 

05 

05 

05 

05 

XT 

o 

o 

05 

05 

05 

05 

05 

05 

o 

05 

05 

05 

05 

05 

05 

0.018 

o 

o 

Y — 1 

h- 

P 

o 

r- 

CO 

CO 

CO 

o 

o 

d 

o 

o 

o 

d 

0 

_c 

o 

o 

05 

d 

o 

d 

CM 

d 

ID 

d 

O 

d 

o 

05 

d 

ID 

o 

o 

Tf 

^r 

ID 

o 

CD 

h- 

o 

d 

CM 

CO 

CO 

Is- 

o 

o 

Y— 

05 

T— 

o 

CO 

o 

o 

CO 

O 

p 

ID 

CM 

Y— 

CO 

CO 

CO 

1 — 

CO 

CO 

o 

o 

C5 

p 

o 

£ 

CO 

CD 

05 

o 

'0- 

ID 

O 

ID 

o 

O 

o 

r- 

CM 

ID 

* 

CD 

CO 

h*. 

o 

o 

CM 

CD 

05 

05 

CM 

CD 

O 

h* 

05 

ID 

00 

CD 

CO 

r- 

o 

N* 

T— 

CO 

h- 

O 

O 

CM 

t- 

T~ 

Y“* 

W 

CD 

CD 

p 

o 

CD 

O 

o 

CD 

CO 

o 

CD 

• 

CD 

CD 

X 

X 

CD 

o 

O 

O 

O 

o 

d 

o 

O 

o 

m 

h- 

CD 

O 

w 

w 

CD 

05 

00 

o 

Si 

ID 

-o- 

s 

5 

ID 

ID 

c 

« 

• 

• 

Yt 

CD 

Y— 

00 

K 

s 

o 

o 

o 

CO 

CD 

O 

ID 

CO 

CD 

h- 

Y~* 

m 

CD 

CD 

o 

CO 

CO 

CO 

O 

o 

■ 

O 

05 

CO 

h* 

05 

o 

CO 

CO 

CO 

o 

w 

ID 

ID 

X 

CO 

CO 

CO 

O 

O 

O 

p 

O 

O 

w 

CO 

CO 

CO 

w 

ID 

ID 

CD 

1"- 

M* 

Y— 

O 

o 

O 

o 

Y~ 

p 

p 

05 

•M- 

05 

CM 

CO 

05 

05 

O 

h** 

ID 

CD 

o 

o 

o 

ID 

CO 

d 

O 

h*. 

O 

CO 

ID 

CD 

CO 

CD 

CO 

CM 

CO 

CO 

CO 

CO 

p 

T". 

d 

T 

d 

<0 

CO 

ID 

CO 

s 

CO 

CO 

CO 

CO 

o 

o 

O 

ID 

'M* 

CO 

CO 

W 

CO 

CO 

CO 

CO 

■_ 

O 

O 

O 

o 

CO 

CO 

CO 

CO 

CO 

CO 

CD 

ID 

CO 

ID 

ID 

05 

ID 

05 

CM 

ID 

d 

d 

o 

o 

10 

05 

3 

O) 

h- 

o 

N- 

CM 

00 

CM 

00 

o 

^1* 

3 

o 

d 

d 

o 

o 

CO 

CD 

CM 

ID 

CO 

ID 

CO 

ID 

CM 

Y — 

ID 

O 

ID 

CO 

ID 

r- 

CO 

Is- 

I’*"* 

CO 

r-. 

O 

05 

o 

d 

CM 

CO 

CM 

CO 

c 

CD 

CO 

CO 

* 

O 

* 

O 

id 

ID 

Y“* 

o 

CD 

W 

CO 

rj 

x 

CO 

o 

O 

CO 

o 

Y— 

o 

CD 

CO 

w 

w 

CO 

ID 

ID 

CO 

ID 

00 

ID 

CD 

ID 

05 

CO 

h- 

o 

o 

'M; 

o 

13 

m 

O 

T” 

CO 

CM 

fe 

o 

Y~ 

CM 

o 

w 

TT 

CO 

N- 

CO 

CO 

CD 

CD 

o 

o 

o 

T— 

CO 

00 

00 

CO 

O 

O' 

CO 

05 

CD 

O 

CD 

CM 

’ 

05 

CO 

Y — 

h- 

** 

CO 

CO 

Y— 

CO 

CO 

CM 

CM 

* 

W 

o 

05 

CM 

o 

CO 

CO 

/X 

CO 

CO 

CO 

£ 

o 

o 

o 

CO 

CO 

o 

CO 

CO 

w 

CO 

CO 

CO 

m 

o 

o 

CO 

p 

O 

d 

05 

CO 

N. 

CO 

CO 

CO 

CM 

CO 

CM 

CO 

00 

ID 

ID 

ID 

CO 

CD 

o 

o 

05 

p 

CO 

CM 

CM 

p 

05 

CO 

CD 

CO 

CO 

CM 

o 

£ 

CO 

CD 

CO 

CM 

ID 

h- 

d 

d 

d 

d 

d 

05 

O 

CD 

CM 

CD 

O 

Y— 

Y — 

h- 

<8 

CO 

h- 

f- 

o 

CO 

Y“ 

h" 

ID 

CM 

d 

CO 

CM 

d 

d 

o 

CD 

CM 

o 

CD 

CD 

CO 

CM 

CD 

O 

CD 

€0 

O 

o 

£ 

CD 

CD 

CO 

fX 

CD 

o 

O 

CM 

h- 

CD 

05 

CD 

CD 

CO 

w 

CD 

CO 

ID 

CD 

CO 

O 

Y— 

O 

o 

T5 

O 

CD 

CD 

o 

YT“ 

h- 

Y— 

o 

d 

d 

3 

Y“ 

O 

T— 

r- 

ID 

05 

"M- 

CO 

O 

5 

CM 

CD 

o 

O 

o 

o 

X 

CO 

o 

CM 

CO 

O 

Y— 

Y— 

o 

K 

CM 

CM 

o 

o 

CO 

CO 

CO 

w 

p 

d 

p 

d 

CO 

CD 

O 

X 

CO 

CO 

o 

O 

o 

CO 

CD 

O 

w 

CO 

(X 

CO 

CM 

o 

d 

CO 

ID 

w 

CO 

r^ 

CM 

ID 

o 

ID 

o 

CD 

d 

d 

p 

ID 

Tf 

ID 

Y“* 

Y~ 

5 

d 

d 

o 

o 

O 

o 

h- 

2 

Is- 

o 

o 

CO 

h- 

'O* 

d 

d 

d 

CO 

CO 

o 

0 

CD 

CO 

CD 

p 

o 

CO 

Is- 

H 

CD 

CO 

CD 

Q 

CM 

CD 

10 

CD 

CO 

CD 

U) 

o 

Y— 

w 

CD 

CO 

CD 

T- 

ID 

o 

o 

o 

o 

05 

d 

d 

15 

O 

o 

o 

£ 

h- 

00 

£ 

£ 

d 

d 

d 

*£ 

o 

O 

CO 

d 

o 

d 

8 

o 

o 

c 

o 

CO 

ID 

CD 

Y“ 

Y— 

CM 

CO 

ID 

CD 

Y— 

o 

Y— 

CM 

CO 

ID 

CD 

Y— 

e 

E 

E 

E 

E 

E 

E 

E 

E 

E 

o 

I 

E 

E 

E 

E 

E 

E 

m 

m 

0 

0 

cd 

0 

cs 

0 

cd 

0 

(h 

0 

0 

m 

0 

0 

0 

d> 

0) 

05 

05 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

h- 

h- 

H 

H 

h- 

H 

f- 

H 

h- 

h- 

Q 

H 

H 

H 

H 

H 

H 

Goals  /  weighting  Complete  goal  rollup  Bins  1-10 

Control 


F 

05 

T» 

CO 

Is- 

05 

L. 

— i 

m 

o 

05 

co 

05 

_ / 

tn 

CO 

Is- 

CO 

Is- 

Is- 

05 

Mi 

O) 

05 

05 

05 

05 

05 

o> 

05 

05 

05 

05 

05 

o 

05 

05 

05 

05 

05 

o 

w 

-c 

o 

d 

o 

d 

O 

o 

o 

o 

00 

O 

o 

Is- 

T“ 

Tt 

05 

00 

CO 

CM 

05 

o 

Is- 

O 

o 

i— 

o 

CO 

d 

o 

o> 

O) 

CM 

CO 

CM 

Is- 

05 

o 

00 

CO 

Is- 

00 

05 

CO 

CD 

O 

05 

00 

O 

in 

O 

o 

O 

O 

CM 

in 

O 

o 

o 

O 

o 

o 

o 

o 

d 

o 

00 

05 

CM 

00 

CM 

N* 

05 

m 

CM 

o 

CO 

00 

CM 

T — 

00 

CO 

CM 

05 

o 

00 

CM 

CO 

CM 

i— 

in 

Is- 

o 

CO 

o 

o 

d 

O 

O 

o 

d 

o 

o 

o 

Is- 

T_ 

CM 

05 

CM 

O) 

CM 

o 

CO 

00 

00 

CM 

T— 

CM 

CO 

Is- 

O 

CO 

00 

in 

Is- 

00 

Is- 

o 

00 

o 

O 

o 

O 

d 

o 

d 

d 

d 

d 

o 

CO 

in 

in 

00 

in 

CM 

00 

CM 

o 

O) 

CO 

in 

o 

in 

CO 

h- 

CM 

CO 

T— 

CO 

CO 

CM 

in 

T“ 

05 

00 

o 

o 

r* 

O 

o 

d 

d 

d 

o 

o 

d 

in 

T— 

in 

O) 

in 

h- 

T- 

LO 

in 

Is- 

o 

m 

o 

00 

CO 

o 

05 

r* 

Is- 

O 

05 

CO 

'M- 

r-. 

in 

CM 

CM 

Is- 

o 

CO 

T— 

co 

d 

d 

o 

o 

d 

d 

05 

T— 

'M- 

m 

Is- 

00 

CM 

00 

00 

CO 

co 

CO 

05 

CM 

in 

00 

m 

Is- 

o 

Is- 

o 

CO 

LO 

T“ 

T— 

CM 

1— 

T— 

o 

T- 

o 

d 

o 

CO 

d 

o 

o 

CO 

CO 

m 

CO 

in 

CO 

CO 

CO 

i— 

o 

CO 

CM 

CM 

o 

i— 

CO 

in 

CO 

in 

'M- 

00 

Is- 

CM 

in 

00 

05 

Is- 

CO 

o 

o 

O 

o 

T*“ 

o 

o 

d 

d 

d 

d 

d 

CM 

m 

CM 

CO 

T- 

o 

00 

CD 

00 

00 

05 

O 

CO 

CO 

05 

05 

i — 

00 

CO 

O 

CM 

o 

o 

T— 

i — 

O 

o 

o 

o 

CO 

o 

o 

d 

T— 

in 

CM 

CO 

CO 

o 

CO 

CM 

CM 

00 

00 

CM 

LO 

00 

05 

CO 

00 

05 

o 

CO 

o 

CO 

d 

d 

o 

o 

o 

d 

d 

o 

T— 

CM 

CO 

in 

CO 

E 

E 

E 

E 

E 

E 

03 

CO 

03 

03 

CO 

03 

CD 

CD 

CD 

© 

© 

© 

h- 

H 

H 

h- 

H 

CO 

CO 

0) 


O 

o 

2 

o 

c 

o 

o 

o 

o 

</> 

Bayesian  projections  for  competition  Bins  1-10 

Team  1 


E 

T— 

T— 

T— 

T~ 

*- 

T- 

E 

T— 

T- 

T- 

T— 

T— 

v- 

E 

T— 

■»— 

T— 

T— 

T— 

T“ 

13 

in 

in 

CO 

O 

o 

o 

© 

© 

© 

sz 

-C 

-C 

o 

o 

o 

o 

o 

o> 

CO 

o 

LO 

o 

o 

co 

m 

o 

T— 

CO 

o 

o 

03 

CM 

o 

CD 

00 

T“ 

CO 

00 

T“ 

o 

CO 

o 

to 

00 

03 

CO 

o 

CO 

h- 

o 

o> 

o 

i 

LU 

CM 

to 

o 

• 

o 

LO 

o 

o 

03 

T“ 

T— 

CM 

CSi 

til 

o 

q 

o 

Is- 

CD 

! — 

LO 

o 

o 

CM 

q 

o 

d 

o 

O 

h- 

T- 

o 

o 

d 

d 

o 

d 

d 

cm 

d 

d 

o 

d 

o 

o 

o 

o 

o 

o 

o 

03 

o 

o 

o 

o 

o 

o 

03 

o 

o 

o 

o 

o 

o 

00 

o 

o 

o 

o 

o 

CO 

00 

o 

o 

o 

o 

o 

T— 

CO 

o 

o 

o 

o 

o 

h- 

to 

CO 

T_ 

o 

to 

CD 

CO 

o 

o 

LO 

o 

o 

■r- 

o 

d 

d 

o 

h- 

o 

o 

o 

o 

o 

r- 

o 

o 

o 

o 

o 

CM 

h* 

o 

o 

o 

o 

o 

CD 

CO 

00 

T— 

CO 

CO 

CD 

T“ 

o 

o 

CD 

o 

o 

o 

o 

o 

d 

CO 

o 

o 

CO 

o 

o 

C- 

CD 

o 

o 

o 

o 

CO 

CD 

o 

o 

Is- 

o 

o 

CM 

CM 

CO 

o 

CM 

CO 

CO 

ID 

03 

T-* 

03 

CO 

o 

T— 

1 — 

o 

T— 

o 

CO 

o 

o 

o 

o 

o 

CD 

CM 

o 

d 

d 

d 

d 

d 

o 

in 

o 

CO 

o 

03 

o 

in 

o 

CM 

o 

CM 

o 

in 

o 

03 

in 

o 

m 

o 

CO 

CD 

CD 

o 

o 

Tf 

CO 

o 

o 

CM 

o 

CD 

CM 

r- 

T“ 

T — 

T— 

t 

o 

T — 

o 

1 

CO 

o 

tn 

* 

o 

T — 

LU 

o 

o 

o 

1 — 

o 

CM 

o 

CM 

03 

o 

o 

o 

© 

CM 

o 

Is- 

o 

o 

T— 

d 

d 

d 

c 

d 

d 

d 

c 

d 

o 

CM 

s 

5 

o 

o 

▼— 

o 

o 

o 

** 

o 

o 

LD 

o 

o 

o 

o 

o 

CO 

o 

o 

o 

o 

LO 

1 

lu 

CM 

o 

CM 

o 

in 

o 

o 

LO 

o 

o 

CO 

o 

CO 

o 

h- 

to 

o 

CD 

o 

CO 

o 

CD 

CO 

o 

CO 

o 

CO 

o 

in 

CO 

o 

tn 

o 

O 

03 

03 

CD 

Is- 

CD 

o 

o 

CO 

CD 

CO 

CO 

o 

LO 

1 

LU 

— ; 

* 

LU 

o 

CM 

to 

o 

CO 

o 

O 

o 

q 

CD 

CD 

o 

CD 

o 

o 

o 

o 

o 

CM 

o 

CM 

d 

d 

d 

d 

d 

CD 

d 

T“ 

CM 

CM 

o 

o 

to 

o 

o 

CM 

00 

o 

o 

Is- 

o 

o 

CM 

h- 

o 

o 

tn 

o 

o 

03 

to 

CM 

CO 

T — 

o 

to 

03 

CM 

03 

CD 

I 

LU 

00 

03 

03 

o 

q 

CO 

CM 

o 

o 

T“ 

d 

o 

o 

o 

o 

03 

d 

d 

d 

d 

CD 

T“ 

CO 

Is- 

03 

to 

CM 

o 

Tt 

CO 

o 

o 

CO 

CO 

CD 

T*** 

CO 

CD 

o 

03 

03 

CM 

Is- 

CM 

T— 

CO 

CM 

CM 

CO 

m 

CO 

CM 

CO 

o 

03 

T— 

Is- 

m 

CD 

CO 

h- 

CO 

03 

03 

CD 

CO 

00 

O 

CO 

CD 

CM 

o 

CO 

o 

03 

03 

o 

03 

03 

03 

q 

03 

03 

CD 

03 

CD 

N. 

03 

03 

03 

03 

Is- 

03 

03 

03 

d 

d 

03 

03 

03 

03 

03 

03 

03 

03 

d 

h- 

03 

03 

d 

d 

d 

d 

d 

d 

d 

d 

d 

o 

o 

d 

o 

o 

o 

t— 

CM 

CO 

to 

CD 

CO 

T— 

CM 

CO 

tn 

CD 

*r- 

to 

T— 

CM 

CO 

to 

CD 

1— 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

CD 

© 

H 

H 

1- 

H 

H 

f- 

H 

f- 

h- 

H 

H 

H 

H 

h- 

h- 

H 

H 

I- 

h- 

E  1" 

3 

<0 

o 

0 

x: 

o 


E 

3 

CO 

o 

0 


o 

o 

00 

00 

o 

T— 

CD 

o 

o 

(30 

CM 

o 

h- 

00 

CD 

in 

CO 

N- 

r- 

CD 

CO 

CD 

to 

CD 

o 

^r 

CO 

cd 

o 

o 

o 

o 

T— 

00 

o 

in 

00 

h- 

o 

o 

o 

o 

r-. 

CM 

o 

d 

o 

o 

o 

o 

CM 

o 

d 

o 

o 

o 

o 

d 

d 

o> 

o 

o 

o 

o 

o 

o 

O) 

o 

o 

o 

o 

o 

o 

00 

o 

o 

o 

o 

o 

00 

o 

o 

o 

o 

o 

00 

TT 

o 

o 

CD 

o 

CO 

o 

o 

o 

o 

o 

h- 

o 

o 

o 

o 

o 

"I— 

h- 

o 

o 

o 

o 

o 

CD 

CO 

CD 

Y- 

CD 

o 

o 

o 

o 

o 

o 

d 

CO 

o 

o 

CO 

o 

o 

00 

CD 

o 

o 

CM 

o 

o 

CM 

h- 

00 

o 

CO 

CD 

CM 

N- 

y— 

o 

o 

CD 

T— 

d 

o 

CO 

o 

o 

d 

d 

in 

o 

h* 

LO 

o 

CD 

o 

m 

o 

"St 

in 

o 

CO 

o 

o 

00 

o 

T— 

N. 

CD 

1— 

o 

CD 

Y~ 

T— 

CO 

CD 

* 

CM 

Y- 

CM 

1 

in 

CM 

o 

o 

o 

CM 

00 

CO 

o 

o 

o 

*/) 

in 

in 

D- 

c 

o 

d 

d 

n 

o 

d 

d 

in 

CO 

o 

o 

N* 

o 

o 

o 

o 

o 

Y— 

o 

o 

o 

CD 

CM 

CM 

o 

Y — 

o 

o 

o 

o 

d 

o 

CO 

o 

CO 

CO 

o 

Y- 

o 

CO 

o 

CD 

CD 

o 

o 

o 

00 

h- 

CO 

h- 

Tt 

oo 

LO 

CD 

T— 

T— 

CD 

CM 

o 

o 

"I — 

o 

o 

o 

o 

o 

o 

o 

d 

d 

d 

d 

d 

CM 

Y — 

o 

o 

CD 

o 

o 

CM 

o 

o 

CO 

o 

o 

CM 

o 

CO 

CD 

CO 

CD 

CD 

CD 

o 

CO 

o 

T— 

CO 

T— 

o 

o 

o 

o 

d 

d 

o 

d 

o> 

Y_ 

Y- 

T- 

CM 

CM 

T“ 

CD 

o 

o 

N- 

o 

CO 

K 

CD 

CD 

CD 

CD 

CO 

o 

in 

CD 

LO 

00 

o 

CD 

o 

CO 

LO 

o 

1 — 

CD 

CD 

5 

CD 

CD 

CO 

CM 

CD 

00 

00 

r^ 

00 

CD 

00 

h- 

cd 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

<D 

d 

d 

d 

o 

d 

d 

o 

d 

o 

o 

o 

o 

CD 

1 — 

CM 

CO 

in 

CD 

T- 

1— 

1 — 

CM 

CO 

LO 

CD 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1- 

H 

h- 

1- 

1- 

h- 

1- 

h- 

H 

h- 

h- 

1- 

1- 

Competition  results  (participants): 


3 

m 

XL 

8 

SZ 

o 


(M  CO  CM  eg 

o  Q  d  d 


o  o  o  o  o  o  o 


co  o  o  o  o  o  o 


r-  o  o  o  o  o  o 


CD  O  O  O  O  O  00 

d 


Oioooo^oo 

t!  0  , 

10 


CD 


o  o  oo  o  co  o 

d  d 


CO  o  o  o  o  o  o 


CM  O  O  O  O  O  O 


O  O  O  CO  o  o 

o 


0 

A 

o 


r-{MCOlT)0i- 

E  E  E  E  E  E 

0  0  0  0  0  0 

0  0  0  0  0  0 

I-  H  H  H  H  H 


£0 


O 


O  O  CO  O  O  O  CO 

T“  o  o 


01  o  o  o  o  o  o 


CO  h-  o  o  o  o  o 

d 


I^OOOOOO 


<0  O  O  O  O  O  O 


0  in  co  h-  o  o  h- 
TT  do  od 


«0 

c 

s 


<0-0  0  0  0  0  0 


gCOOON-OOO 

o  o 

0 

0 

w 

** 

qCMOOCOOCOO 

V  do 

i 

iSr-oOO^OO 


0 

Q. 

>* 

CO 

S 


€0 

o 


t-DJCOLOCDt- 

E  E  E  E  E  E 

CO  CO  CO  CO  CO  CO 

0  0  0  ©  ©  © 

I —  I —  I —  I —  I —  J — 


03 

m 

X 

o 

0 

sz 

o 

oo^rooo^- 
o  o 


o>  o  o  o  o  o  o 


00  ID  O  O 

d 


h«-  o  o  o 


CO  o  o 

d 


o  in  ID  to  id 

2  CO  o  T- 


10 

c 

s 


o  o  o 

O  LD  O 

d 

CO  o  o  id 


£  O 

O 

£ 

2  CM  o  o  N. 

c  d 

o 

v  -  -t  ^  ° 

0i  do 


o 

o 

CM 

CD 

t> 


T-  CM  CO 

E  E  E 

0  0  0 

H  h  H 


Team  5  10000000 

Team  6  0  0.3  0.1  0  0.05  0.4  0  0.15 

Team  10  0.4  0  0  0  0.2  0  0  0 


S  (score)  Bins  1-10 

12  3  4  5  6  7  8  9  lOchecksum 

Teaml  0.1  0  0  0  0.07  0.65  0  0  0  0.18  1 

Team  2  0.25  0  0  0  0  0  0  0  0  0.75  1 

Team  3  0.8  0  0.1  0  0.1  0  0  0  0  0  1 


N  O  IT) 
O  CD 


O  O  CO 
O 


OWN 

o  o 


o  o  o  o  o  r- 

i-  CO  CO 


0*0  0  0  0  00 


co  lo  o  o  o  m  o 
co  2 


o  o  o  o  o  o 


to  h-  O  O  O  00  o 

O  CM 

o  o 


u>  in  in  in  o  in  in 

co  ^  o  ^  co 

00  CM  t-  w  o 


^  o  in  o  o  o  o 
o 


00  O  O  ID  O  Is*-  O 
T-  O 

CO  r-i 


CM  O  O  00  O  CO  O 

w  o 


00  00  o 

CM  CM 


o 

LO  CD  T- 

E  E  E 

(d  cd  d 
0)  Q)  CD 


*i-  CM  CO  LO  CD  *1“ 

E  E  E  E  E  E 

cd  cd  cd  (d  cd  (d 

CD  0  CD  CD  <D  CD 

I —  I —  I —  1 —  K —  I — 


Goals  /  weighting  Complete  goal  rollup  Bins  1-10 

Control 


3 

CO 


o 


o 

00 

CM 

00 

CM 

T— 

O 

€0 

o 

O 

CM 

CM 

<0 

d 

o 

o 

in 

d 

d 

o 

o 

O) 

o 

o 

o 

o 

o 

o 

00 


o  o  o  m  o 

to 

o 


O  O  O  O  O  00 

o 

o 


to 


't 

d 


o  o  o  ^  h- 

00  r- 

9  d 
o 


in  in 

m 

in 

in 

h- 

CM 

05 

T“ 

i— 

^r 

K 

05 

o 

o> 

T — 

O 

O 

d 

o 

o 

d 

d 

d 

d 

**  o 


in  oo  o  h- 

i-o  t- 

§  o  o 
d 


cm 

r- 


00  o  o 


€M  O  O 


in  oo 

r  « 

in  o  o 

T“.  o  o 
o 


Tf  O  05  o 
h-  o 


00  CM  CM 

co  o)  co  ^ 

-  9  o  d  ^ 

o  o  o  o 


i — 

CM 

CO 

m 

<0 

i- 

E 

E 

E 

E 

E 

E 

ed 

cd 

cd 

<d 

m 

cd 

o> 

CD 

o 

o 

o> 

o 

1- 

H 

H 

H 

H 

H 

CO 

10 

d 

o 

£ 

o 

o 

£ 

o 

w 

4-* 

£ 

O 

o 


o 

o 

</> 


difference  squared  between  actual  results  and  bayesian  projections  for  competition 
far  right  gives  rms  error 

Team  1  Bins  1-10 


CO 

T— 

in 

in 

a 

CO 

in 

in 

CM 

o 

r- 

o 

CO 

o 

T-” 

00 

CM 

a> 

T— 

o 

O) 

CM 

LO 

o 

2 

CM 

CO 

CO 

o> 

2 

CO 

CO 

1“ 

o 

T- 

CO 

2 

CO 

o 

i— 

O 

CM 

CM 

CO 

CO 

o 

i — 

0) 

o 

o 

o 

o 

o 

o 

0) 

d 

o 

o 

O 

o 

o 

<5 

O 

o 

o 

o 

o 

<A 

(A 

(A 

E 

E 

E 

i- 

o 

CO 

CO 

o 

Si! 

o 

i— 

o 

CO 

CM 

o 

CM 

00 

CO 

© 

CO 

CO 

o 

CM 

o 

r- 

o 

o 

o 

in 

T- 

a 

■j— 

o 

O 

o 

CO 

CO 

o 

o 

o 

o 

o 

o 

o 

CM 

o 

o 

O 

o 

o 

o 

o 

o 

d 

o 

o 

o 

o 

d 

o 

o 

o 

o 

d 

o 

o 

d 

d 

o 

o 

o> 

o 

o 

o 

o 

o 

o 

o>  o 

o 

o 

o 

o 

o 

o>  o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

d 

d 

d 

o 

o 

o 

o 

d 

o 

o 

o 

d 

00 

o> 

o 

o 

o 

T— 

o 

oo  o> 

o 

o 

o 

T- 

CO 

00  CD 

o 

o 

o 

1_ 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

o 

o 

o 

d 

o 

o 

o 

d 

o 

o 

o 

d 

d 

o 

o 

N- 

O 

o 

o 

o 

o 

o 

h-  o 

o 

o 

o 

o 

CM 

h-  o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

o 

o 

o 

o 

d 

d 

o 

o 

o 

d 

d 

o 

o 

d 

d 

o 

CO 

CD 

o 

o 

o 

i^- 

to  CD 

o 

o 

o 

CO 

to  o> 

o 

o 

o 

CO 

o 

o 

o 

o 

CM 

CO 

o 

o 

o 

o 

CO 

o 

o 

o 

i- 

o 

o 

o 

o 

O 

T“ 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

d 

d 

o 

o 

o 

o 

o 

o 

o 

d 

o 

o 

o 

o 

o 

o 

o 

in 

o 

in 

00 

o 

00 

CM 

o 

LO  O 

o> 

00 

o 

CM 

CM 

o 

in  o 

in 

LO 

o 

00 

Si! 

o 

o 

o 

o 

CM 

CM 

o 

o 

o 

CM 

T" 

CM 

o 

O 

o 

o 

o 

o 

o 

o 

o 

O 

1 

O 

o 

o 

o 

O 

1 

O 

o 

o 

o 

o 

d 

d 

d 

o 

d 

O 

<A 

C 

s 

d 

o 

o 

d 

d 

o 

(A 

c 

£ 

d 

d 

d 

d 

o 

o 

CO 

o 

CD 

o 

o 

T— 

CO 

o 

CD 

o 

o 

CO 

o 

G) 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

d 

d 

o 

o 

o 

d 

d 

d 

o 

o 

o 

o 

d 

d 

d 

d 

o 

CO 

o 

o 

CO 

o 

CO 

o 

00  o 

o 

o 

h- 

o 

CO  o 

o 

CO 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

O 

o 

o 

o 

d 

o 

d 

d 

d 

d 

d 

o 

o 

o 

o 

d 

d 

d 

d 

o 

CM 

-i— 

o 

o 

V“ 

00 

o 

CM  i- 

o 

o 

o 

00 

o 

CM  O 

o 

o 

o 

00 

O 

o 

CO 

O 

o 

o 

O 

o 

CO 

o 

o 

o 

o 

o 

CO 

o 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

i-  CO 

T— 

CM 

CD 

CD 

T“  -sf- 

in 

CD 

LO 

N- 

CM 

CO 

rj- 

CO 

CO 

f^- 

CO 

CO 

o 

1 — 

CO 

00 

LO 

CO 

O 

CO 

CM 

CO 

00 

00 

o 

CM 

CO 

LO 

CM 

o 

CO 

CO 

CD 

o 

CM 

O 

O 

r^ 

in 

o 

o 

CO 

o 

o 

o 

d 

o 

d 

o 

d 

o 

d 

o 

d 

d 

d 

d 

d 

d 

T- 

CM 

CO 

LO 

CO 

o 

CO 

CM 

CO 

in 

CO 

O 

,i — 

in 

T— 

CM 

CO 

in 

CO 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

03 

03 

03 

03 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

H 

1— 

H 

H 

H 

1“ 

H 

h- 

h- 

1- 

H 

H 

h- 

1— 

H 

1- 

H 

Team  10  0.727  0.000  0.000  0.000  0.022  0.028  0.000  0.000  0.000  0.271  0.324 


Team  6  Bins  1-10 

1  2  3  4  5  6  7  8  9  lOrms  error 

Teaml  0.611  0.006  0.000  0.000  0.020  0.169  0.000  0.009  0.000  0.043  0.293 

Team  2  0.572  0.000  0.000  0.001  0.005  0.000  0.000  0.000  0.000  0.430  0.317 

Team  3  0.252  0.030  0.023  0.006  0.008  0.000  0.000  0.000  0.000  0.000  0.179 


00  00  o 

CM  CO  CM 

O  -r-  CO 

odd 


CM  O  CO 
O  O  CD 
O  O  CM 

odd 


CO  CM  r-  CO  ID  N 
•  ONCOCMNr- 
2  CO  r*  CM  O  CM  CO 

Jjddddod 

m 

£ 

O  CO  CD  O  CM  G> 
r  Tt  CO  O  O  CO  CD 
O  O  O  O  O  CM 

d  d  d  d  d  o 


ooo  0)000000 
ooo  oooooo 
ooo  oooooo 

odd  ddddod 


O  'i-  o 
ooo 
ooo 

ooo 


00  O  O  O  O  'T-  o 
OOOOOO 
OOOOOO 

ddddod 


ooo  h-oooooo 
ooo  oooooo 
ooo  oooooo 
ooo  ddddod 


O  N  CO  CD  O)  O  00  O  h-  LO 

O  O  CM  CO  O  ID  O  O  CM 

OOO  t-  O  O  O  O 

odd  dodddd 


O  00  CM  OlOO^rONCM 
O  O  CM  T-  CM  O  00  O  Is-  CM 

OOO  »  O  CM  CM  O  ^  O 

odd  m  dddodd 


r- 


o 

CD 

o 

**  o 

y— 

CD 

o 

CD 

o 

o 

CM 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

o 

o 

CO  o 

o 

CO 

o 

h- 

o 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

o 

d 

d 

d 

d 

d 

d 

o 

00 

o 

CM  1- 

o 

o 

o 

CO 

o 

o 

o 

o 

o 

o 

CO 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

ID 

h- 

CD 

ID 

o 

ID 

CM 

o 

CD 

K 

ID 

CO 

o 

CO 

0> 

o 

CM 

CD 

o 

CM 

o 

T- 

CD 

d 

d 

d 

d 

d 

d 

d 

d 

d 

o 

o 

o 

ID 

CD 

1— 

T— 

CM 

CO 

ID 

CD 

r* 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

cd 

<d 

cd 

cd 

cd 

cd 

cd 

cd 

cd 

cd 

<d 

o 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

H 

H 

H 

H 

H 

H 

H 

H 

difference  squared  between  actual  results  and  team  projections  for  competition 
far  right  gives  rms  error 

Team  1 

Bins  1-10 


s 

CM 

Is- 

CO 

CM 

CO 

YT“ 

00 

LO 

o 

CO 

Is- 

CM 

1^ 

CM 

o 

CO 

to 

CM 

CO 

LO 

00 

o 

00 

CO 

ka 

CO 

00 

CD 

o> 

o 

2 

T- 

OJ 

T~ 

CO 

T— 

1— 

2 

T— 

CM 

CM 

CM 

1 — 

CM 

2 

T“ 

CM 

T— 

CM 

o 

CM 

k. 

Q> 

o 

o 

o 

o 

o 

d 

k. 

G> 

o 

o 

d 

o 

o 

o 

k. 

<D 

o 

d 

o 

o 

o 

o 

W 

0) 

w 

E 

E 

E 

k. 

k. 

k. 

o 

T— 

LO 

Is- 

CM 

N- 

o 

o 

o 

T— 

00 

T— 

o 

o> 

o 

o 

00 

1— 

1— 

CO 

T~ 

CO 

CO 

o 

o 

o 

CO 

T— 

CO 

CM 

o 

o 

o 

LO 

CM 

CO 

o 

o 

o 

r^ 

o 

o 

o 

o 

T— 

o 

o 

o 

o 

CM 

o 

CO 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

o 

o 

o 

o 

d 

o 

o 

o 

d 

o 

o 

o 

O)  CM 

o 

CM 

Is- 

LO 

o>  o 

T- 

o 

o 

o 

CM 

o>  t- 

o> 

CD 

o 

Is- 

o 

o 

T- 

o 

T— 

T- 

o 

o 

CD 

o 

o 

o 

CO 

o 

O 

o 

CM 

CO 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

O 

d 

o 

o 

o 

o 

o 

o 

d 

o 

o 

o 

d 

o 

o 

o 

d 

d 

o 

00  o 

-r— 

CM 

CO 

i^ 

CM 

00  CO 

Y — 

o 

o 

o 

00  CO 

o 

o 

CM 

o> 

o 

O 

T" 

CM 

o 

Y — 

o 

o 

CO 

o 

o 

LO 

o 

o 

o 

o 

CO 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

o 

o 

o 

d 

d 

o 

d 

d 

d 

o 

o 

d 

N- 

LO 

00 

CM 

CM 

<1— 

Is- 

N. 

h- 

o 

Is- 

h- 

o 

o 

O 

o 

T— 

CM 

T“ 

o 

o 

T— 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

T— 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

o 

CO 

LO 

CO 

LO 

o 

CO 

CO 

o 

LO 

00 

CM 

CM 

o 

CO 

o 

00 

o 

o 

CD 

CO 

CM 

1— 

LO 

o 

o 

CD 

o 

o 

o 

o 

CM 

CM 

o 

CM 

o 

o 

CM 

T“ 

O 

o 

o 

o 

o 

O 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

o 

o 

o 

d 

d 

o 

d 

o 

o 

o 

o 

d 

to  ^ 

o 

Y — 

o 

CM 

o 

LO  i- 

o 

T- 

r- 

O 

o 

LO  CD 

CM 

LO 

o 

i — 

CM 

o 

o 

o 

YP* 

o 

o 

o 

o 

o 

o 

T — 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

o 

o 

1 

1— 

o 

o 

o 

o 

o 

o 

1 

1“ 

o 

o 

o 

o 

o 

O 

d 

d 

o 

o 

d 

d 

(/) 

o 

o 

d 

d 

d 

o 

(/) 

o 

d 

o 

d 

d 

d 

c 

c 

ffl 

s 

CO 

■t— 

LO 

f— 

o 

o 

LO 

Y— 

o 

o 

o 

00 

o 

o 

o 

o 

o 

o 

o 

T — 

r>- 

00 

o 

o 

T — 

o 

o 

o 

CO 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

d 

d 

d 

d 

d 

d 

d 

o 

o 

d 

d 

o 

o 

d 

d 

d 

o 

d 

CO  N- 

LO 

o 

o 

h- 

CO  CO 

CM 

CM 

Tfr 

CO 

o 

CO  o 

CD 

I— 

LO 

o 

o 

i — 

O 

o 

o 

o 

CD 

CM 

T — 

CM 

o 

o 

CM 

00 

o 

o 

o 

o 

o 

o 

o 

o 

O 

O 

o 

o 

o 

o 

o 

o 

O 

o 

o 

o 

d 

d 

o 

d 

d 

d 

o 

d 

o 

o 

o 

o 

o 

d 

o 

d 

d 

d 

CM  LO 

''si' 

i— 

o 

o 

00 

CM  CO 

LO 

CD 

00 

00 

o 

CM  y 

00 

CD 

LO 

o 

Is- 

T - 

o 

o 

o 

O 

o 

CM 

T — 

o 

o 

CD 

CM 

CM 

CM 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

CM 

o 

o 

o 

o 

O 

o 

o 

d 

d 

d 

d 

o 

d 

d 

o 

o 

o 

o 

d 

d 

o 

o 

o 

d 

o 

T-  CD 

CO 

o 

CO 

00 

LO 

Y— 

CM 

o 

CD 

Is- 

T— 

r-  ''sf 

o 

o 

Is- 

CD 

CO 

1 — 

r- 

CM 

o 

CM 

LO 

CO 

LO 

00 

CM 

o 

o 

CD 

Y“ 

CM 

o 

o 

Y— 

00 

Y— 

o 

O 

o 

CM 

LO 

O 

o 

o 

Y— 

CO 

o 

O 

d 

o 

o 

o 

o 

o 

o 

o 

d 

o 

o 

o 

d 

o 

o 

d 

d 

o 

Y — 

CM 

CO 

LO 

CO 

o 

CO 

CM 

CO 

LO 

CO 

o 

LO 

Y — 

CM 

CO 

LO 

CO 

O 

T“ 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

CO 

CO 

CO 

CO 

c0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

CD 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

h- 

H 

H 

h- 

h- 

1- 

H 

H 

h- 

1- 

H 

H 

1- 

h- 

h- 

H 

1- 

h“ 

Team  6  Bins  1-10 

1234  5678  9  10rms  error 

Team  1  0.021  0.002  0.003  0.000  0.001  0.076  0.051  0.023  0.006  0.033  0.147 

Team  2  0.012  0.129  0.020  0.000  0.002  0.000  0.001  0.000  0.004  0.232  0.200 

Team  3  0.196  0.017  0.009  0.000  0.002  0.074  0.029  0.034  0.008  0.000  0.192 


co  cm  ^ 
©  ©  in 

CM  r*  r 

odd 


a>  co  co  oo  id  cm 

^  CM  O)  00  S  N 
<1-  C\J  T-  CM  O 

d  d  o  o  o  d 


T— 

m 

CD 

O  CO 

CM 

o 

CM 

o 

00 

o 

CD 

CD 

T-  ^ 

CD 

o 

o 

o 

o 

o 

o 

o 

o 

CO 

o 

o 

o 

o 

d 

d 

d 

d 

d 

d 

o 

o 

o 

o  o  CM 
o  o  y- 
o  o  r- 

d  o  o 


OOr-OJOr  CO 
O  CM  O  O  O  O 

o  o  o  o  o  o 
o  o  o  o  d  o 


o 

o 

CM 

00 

CD 

co 

© 

*r- 

© 

o 

o 

T~ 

© 

CM 

© 

© 

© 

© 

o 

o 

o 

o 

© 

o 

© 

o 

o 

o 

o 

d 

o 

o 

o 

o 

o 

© 

o 

o 

h* 

ID 

€0 

ID 

00 

r- 

o 

o 

o 

O 

© 

© 

© 

o 

© 

o 

o 

o 

O 

o 

o 

o 

o 

© 

o 

o 

o 

O 

© 

© 

© 

© 

© 

CD 

CD 

CO 

CO 

CO 

00 

© 

00 

o 

o 

CM 

00 

© 

T— 

o 

© 

Y“* 

o 

o 

o 

o 

© 

© 

© 

© 

o 

d 

d 

d 

o 

d 

d 

d 

© 

© 

CM 

CD 

© 

ID 

h- 

ID 

CM 

© 

o 

o 

o 

t- 

T“ 

o 

© 

ID 

T — 

00 

T — ■ 

o 

o 

o 

1 

© 

o 

© 

o 

© 

© 

d 

d 

d 

0 

o 

d 

o 

© 

d 

d 

SO 

r- 


c 

m 


CM 

© 

© 

ID 

00 

CM 

T— 

CO 

T— 

© 

T— 

O 

CO 

i — 

CM 

o 

O 

© 

o 

o 

© 

© 

O 

© 

o 

© 

© 

d 

© 

© 

© 

o 

© 

o 

o 

CO 

N- 

o> 

ID 

© 

CO 

LO 

o 

o 

O 

o 

T — 

CM 

O 

o 

© 

o 

o 

o 

o 

© 

O 

© 

o 

d 

© 

© 

© 

o 

© 

© 

© 

© 

ID 

h- 

© 

CM 

© 

00 

CD 

© 

00 

© 

© 

o 

© 

CM 

0> 

o 

© 

T— 

© 

o 

o 

© 

© 

© 

o 

© 

© 

© 

© 

© 

d 

d 

d 

d 

d 

o 

CO 

CO 

h- 

CM 

h- 

t — 

CO 

o 

o 

CD 

CM 

t — 

N- 

00 

o 

ID 

© 

© 

© 

© 

CM 

© 

© 

d 

d 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

ID 

© 

T~ 

T— 

T“ 

CM 

© 

© 

© 

T — 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

€0 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

© 

H 

fr- 

H 

H 

H 

h 

h- 

H 

H 

H 

*  JJ 

•C  £  & 

.O  CS  o 

1 1  <§  % 
•S«dc 

2  o  § 

'd'S  §)§ 
a  ^  c  a 

f  i  3  •!—)  "*"* 


®  s  5 's 

o  ^  i  w 

^  O  S  tfl 

c  — '  2  <l> 

o  o  ^  o 


3  2  E  <u 

i  1a><8  "S 

s  ja  a  S> 
3  T3  *S  -5 

3>|.SS 

1  Mf 

2  b  k  co 

5  22  ^  .2 

3  s  o  n 

2  g  ’S  s 

J  2  <l>  Q 

9-  o  a  a 

3  r  m  4) 

oj)> 


5  c  ~ 
d  o  <o 

so  £  ’S 
-  o  0) 

*1  I  S3 

^  §  <N 

§ 

a  ■g  a 

§  §  a 

“njs 

H  J  ^ 

r  .§  W 
2  -S  T3 
§ 

g  ^  ^  >> 

ki  m  C  o 

6  ^  \S 

c5  e|  8 

-s  -s  -a  & 

,5  S  W  2 


<D  *3  "-H 

53  (L>  P- 

«ts  *-  g 
So 
2^U 

frp<£  <u 

*C  °  J3 


o  ^  *3 

HJ<s 

I  h  :1 

ts  •.  -o 
S  2  js 
-o  c  x 
<d  <u  pq 

pfi  >  * 

+-*  <u  W3 

o3  P  C 
^  >  O 
«  *0  > 
£>  o  o 
£  0>  ^ 

5  g  g 

’g  ^  & 
S  o  S 
^  SS 

rv  ^  JD 

is  o  w 


177 


intentionally  left  blank 


On 

r- 


Appendix  G:  Each  team’s  methodology  in  assigning  scores 

In  Section  9.1.1  a  synopsis  of  the  methods  used  by  the  teams  in  assigning  scores 
is  offered.  A  description  of  each  of  the  team-specific  methods  is  found  below. 

Method  used  by  Team  1. 

1.  Read  the  Wiki  journal  daily  to  learn  the  best  practices  of  other 
teams  and  avoid  problems  encountered  by  other  teams.  The 
Wiki  Journal,  found  on  the  MASLAB  website  [1],  is  a  journal  in 
which  the  design  teams  update  their  daily  progress. 

2.  Observe  peers  in  the  lab  and  make  comparisons  to  own  design 
and  progress. 

3.  Keep  track  of  how  much  effort  is  put  into  the  robot  as  an 
indication  of  expected  success  in  the  upcoming  contests. 

4.  Use  performance  in  previous  events  as  evidence  in  evaluations 
of  projected  success  in  future  events. 

Method  used  by  Team  3. 

1 .  Observe  peers  in  the  lab  environment. 

2.  Speak  with  the  other  robot  design  teams  to  identify  their  best 
practices  and  problems  encountered. 

3.  Use  performance  in  previous  events  as  evidence  in  evaluations 
of  projected  success  in  future  events. 
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Method  used  by  Teams  5  and  6. 

1.  Observe  peers  in  the  lab  environment. 

2.  Use  performance  in  previous  events  as  evidence  in  evaluations 
of  projected  success  in  future  events. 

Method  used  by  Team  10. 

1.  Observe  peers  in  the  lab  environment.  Question  whether  other 
teams  often  lack  direction  or  consistently  know  what  they  are 
doing?  Ask  whether  teams  are  able  to  build  sensors  on  their 
own?  If  they  can,  whether  they  are  performing  well  above 
average  and  are  expected  to  do  well  in  the  contests.  Ask 
whether  teams  are  using  water-jetting  (i.e.,  a  sophisticated 
manufacturing  method)  for  the  robot  chassis?  If  a  team  is  using 
such  a  method  in  the  manufacture  of  the  robot  chassis,  their 
robot  is  likely  to  be  fairly  sophisticated  overall  and  excellent 
performance  thus  is  expected.  Ask  whether  teams  are 
consistently  making  last  minute  preparations  prior  to  the 
contests?  If  so,  then  poor  performance  will  likely  result. 

2.  Use  performance  in  previous  events  as  evidence  in  evaluations 
of  projected  success  in  future  events. 
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Appendix  H:  Making  the  evaluation  process  easier  for  each 

team 

In  Section  9.1.2  a  synopsis  of  the  what  would  have  made  the  evaluation  process 
easier  is  offered.  A  description  of  what  would  have  made  the  evaluation  process  easier 
for  each  of  the  teams  is  found  below. 

Team  1 

The  capabilities  of  the  robots  were  not  equal  in  each  of  the  contests  (i.e.,  full 
capability  was  not  achieved  until  the  final  contest,  the  Competition).  This  fact  meant  that 
some  of  the  questions  asked  in  the  questionnaire  did  not  apply  until  Exhibition  2  and 
possibly  until  the  Competition.  This  fact  meant  that  the  questionnaires  needed  to  be 
better  suited  to  each  event  such  that  not  only  a  current  but  also  a  fairly  exact  assessment 
of  each  robot  capability  could  be  obtained  by  filling  out  such  surveys. 

Team  3 

Nothing  would  have  made  the  evaluation  process  easier.  The  use  of  surveys  was 
a  good  way  to  evaluate  a  robot’s  overall  capability. 

Team  5 

More  time  should  be  spent  in  filling  out  the  questionnaires.  This  was,  however,  a 
difficult  issue  to  resolve  as  preparation  of  the  robot  for  each  of  the  three  events  was  a 
highly  time  constrained  process. 
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Team  6 


Be  consistent  in  filling  out  the  questionnaires.  That  is,  having  previous 
questionnaire(s)  available  could  assist  in  maintaining  greater  consistency  in  answering 
the  questionnaires. 

Team  10 

Completing  the  first  evaluation  was  difficult.  Since  the  contests  themselves  offer 
the  best  evidence  of  a  robot’s  capability,  the  first  evaluation  was  based  largely  on 
observations  made  in  the  lab  prior  to  Exhibition  1.  This  meant  that  more  time  must  be 
spent  in  making  not  only  the  first  evaluation  but  also  the  follow-on  evaluations.  The 
additional  time  could  have  been  spent  in  discussing  problems  as  well  as  best  practices 
and  plans  of  attack  with  each  of  the  competing  teams  and  in  reading  the  Wiki  journal  [1]. 
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